Wednesday, January 30, 2013
Tuesday, January 29, 2013
Friday, January 25, 2013
HCI Big Data Project Proposal: Is there a Correlation Between Energy Utilization and Google Searches for Green-Power Providers in Texas
HCI Big Data Project Proposal
Nick Goodman
Updated January 30, 2013
I propose to look for correlations between searches for green-power providers and energy utilization in Texas. I will do this by scraping power-grid utilization and processing Google Trends data.
The Energy Reliability Council of Texas -- better known by its acronym ERCOT -- is the Independent System Operator (ISO) for a large portion of Texas. It is one of the ten ISOs in the United States.
Hat tip to the FERC (Federal Energry Regulatory Comission) for the map.
I propose to scrape the ERCOT real-time system conditions website every 15 minutes:
http://www.ercot.com/content/cdr/html/real_time_system_conditions.html
I will be extracting the Actual System Demand (how much power is being drawn from the electricity grid), the Total System Capacity (how much capacity the grid currently has), and Total Wind Output (the generation from Wind turbines). I will also be capturing the other data shown in the screen shots in case it turns out to be interesting.
This data is interesting because it provides insight into energy demand and into wind power generation. I believe that this data can be correlated with several factors:
The data feed is real-time and changes frequenty, though often by small amounts. The data is measured from the electricity grid. I expect short outages in data availability.
To correlate grid utilization data with search trends, I will use Google Trends data. Google Trends exposes the frequency of search terms (you can specify up to five). It can be filtered by geography and time. The data is expressed as a count per day.
Google Search Trends Process
I will scrape the Google search trends either manually (the readings are once per day and are available historically) or using a custom-coded parser or a third-party parser (there is no API for this). The data is available in CSV format.
ERCOT Data Volume
10 numbers and 1 timestmp = (4 bytes) (10) + (8 bytes)(1) = 48 bytes
(48 bytes) * (4 times / hour) (24 hours / day) = 4,608 bytes / day
(4608 bytes / day) (60 days / 2 months) (2 months) = 270 KB
Google Trends Data Volume
5 numbers and 1 timestamp = (4 bytes)(5) + (8 bytes)(1) = 28 bytes
(28 bytes) * (1 time / day) * (60 days / 2 months)(2 months) = 1680 bytes
Total Data Volume
272 KB
Technology Stack
I will develop a J2EE application to scrape and store this data. It will likely run inside of a JBoss application server on a CentOS machine in my office. The data will be stored in a MySQL database. For reliability purposes, I will likely design the process as a two-phase import: (i) download the data and store it on the hard drive and (ii) process the data and put it into the database. This will minimize problems related to bugs in the import code without losing the power of a database.
Estimated Cost
I estimate no cost (other than my time). All of the tools I will use are open-source and freely-available. The machine in question is already built as a CentOS JBoss/MySQL server used for development.
Nick Goodman
Updated January 30, 2013
I propose to look for correlations between searches for green-power providers and energy utilization in Texas. I will do this by scraping power-grid utilization and processing Google Trends data.
The Energy Reliability Council of Texas -- better known by its acronym ERCOT -- is the Independent System Operator (ISO) for a large portion of Texas. It is one of the ten ISOs in the United States.
I propose to scrape the ERCOT real-time system conditions website every 15 minutes:
http://www.ercot.com/content/cdr/html/real_time_system_conditions.html
I will be extracting the Actual System Demand (how much power is being drawn from the electricity grid), the Total System Capacity (how much capacity the grid currently has), and Total Wind Output (the generation from Wind turbines). I will also be capturing the other data shown in the screen shots in case it turns out to be interesting.
This data is interesting because it provides insight into energy demand and into wind power generation. I believe that this data can be correlated with several factors:
- Temperature
- Wholesale Electricity Market Price
- Time of Day
- Day of Week
The data feed is real-time and changes frequenty, though often by small amounts. The data is measured from the electricity grid. I expect short outages in data availability.
Note the Slight Change in Frequency In the Screenshots
To correlate grid utilization data with search trends, I will use Google Trends data. Google Trends exposes the frequency of search terms (you can specify up to five). It can be filtered by geography and time. The data is expressed as a count per day.
Google Search Trends Process
I will scrape the Google search trends either manually (the readings are once per day and are available historically) or using a custom-coded parser or a third-party parser (there is no API for this). The data is available in CSV format.
ERCOT Data Volume
10 numbers and 1 timestmp = (4 bytes) (10) + (8 bytes)(1) = 48 bytes
(48 bytes) * (4 times / hour) (24 hours / day) = 4,608 bytes / day
(4608 bytes / day) (60 days / 2 months) (2 months) = 270 KB
Google Trends Data Volume
5 numbers and 1 timestamp = (4 bytes)(5) + (8 bytes)(1) = 28 bytes
(28 bytes) * (1 time / day) * (60 days / 2 months)(2 months) = 1680 bytes
Total Data Volume
272 KB
Technology Stack
I will develop a J2EE application to scrape and store this data. It will likely run inside of a JBoss application server on a CentOS machine in my office. The data will be stored in a MySQL database. For reliability purposes, I will likely design the process as a two-phase import: (i) download the data and store it on the hard drive and (ii) process the data and put it into the database. This will minimize problems related to bugs in the import code without losing the power of a database.
Estimated Cost
I estimate no cost (other than my time). All of the tools I will use are open-source and freely-available. The machine in question is already built as a CentOS JBoss/MySQL server used for development.
Subscribe to:
Posts (Atom)