Related Articles

URLs in this article:

Probabilistic Assessment of Timing of Access
to the US Retail Power Market

by Anne Ku (May 1997)

Presented in May 1997 at INFORMS Spring 97 Conference in
San Diego, California

Abstract

The transition to a fully competitive retail power market depends on a myriad of factors through a process which is largely political and highly uncertain. Individual states vary widely in terms of the pace of decision making and issues in restructuring. A four month study was conducted to investigate and implement a method of answering the question: "When will we have access to the first retail customer in each state?" Several approaches to predicting the timing of retail access are proposed and assessed with respect to four criteria: simple, systematic, defensible, and acceptable. A pilot study found two of these most suitable: behavioral aggregation of judgment-based group scenario-contingent assessment and the non-judgment based event tracking approaches. The two approaches were implemented to track the timing and likelihood of access on a regular basis.

Rationale for Predicting Timing of Retail Access

The deregulation and restructuring of the giant US electricity supply industry is underway in every state. Once opened up, the US retail power market is estimated to command $200 billion per year in revenues, which is considerably greater than natural gas and other deregulated industries. Whereas the deregulation of the natural gas industry took more than a decade, the power industry is expected to go through the process in half that time. Currently, except for pilot programs implemented in a handful of states, customers are not allowed to choose their electricity supplier. With retail access, customers and electricity suppliers will be able to choose. The question of how soon they will be able to choose or "access" depends on a political process of resolving various issues, agreeing to a date of access, and implementing the final policy elements. The speed of this process varies from state to state.

Enron is the largest independent wholesale power marketer in the US. Retail, i.e. access to the final end user, presents an enormous business opportunity. To tap into this market, Enron requires a method for tracking the timing and likelihood of access. This four month project was undertaken to develop a simple, systematic and defensible approach of predicting timing of deregulation that is acceptable to the user of this information. A simple approach refers to one that could be implemented without detailed involvement of many parties through a process that is transparent. "Systematic" describes a method that is straightforward and could be automated, without too many interventions or exceptions. It is important that these predictions are kept up-to-date. "Defensible" is equivalent to scientifically acceptable. User acceptance may very well depend on communication of the methodology and content.

This study focuses on predicting the timing of start of retail access only, not phase-ins of different market segments. Although methodologies for other dates are discussed, the results pertain to predicting the year that the first retail customer in each state can be accessed.

The Problem

The timing of deregulation refers to the approval, start, and phase-in dates of retail access.

Participants in the restructuring discussions describe the process as one of on-going debate about various controversial issues until a consensus is reached (Raab, 1994). Experience thus far has shown it to be a long and laborious process, one that is highly political as it involves stakeholders with different agendas and degrees of influence. Most states require legislative approval of retail access, i.e. mere regulatory approval is not sufficient. Such restructuring orders typically declare start dates of access and subsequent phase-ins. However, the actual date of implementation depends on the resolution of various issues and the readiness of systems and procedures put in place.

One can view the road to retail access as passing through five stages. The first stage is an inquiry into retail access. Most states are in this stage, except for those which have either not started the inquiries or have decided that retail access is not desirable. The inquiry stage is followed by the development of a policy decision, usually at the regulatory level. Typically policy recommendations and a timetable are taken to the state legislature, where rule-making occurs. At each stage, certainty about the timing of deregulation increases. With each stage, the uncertainty about dates decreases. Table 1 shows an example of the degree of certainty one can expect from state progress. The probability ranges reflect the author’s assessment based on interviews with various experts.

Several factors complicate this task of predicting the timing of deregulation. To investigate suitable approaches, one needs to understand the process, identify willing well-informed experts, and aggregate individual judgments. The task of time prediction, especially of an outcome of a political process, is far more difficult than predicting the outcome (winning candidate) of an election or the verdict (guilty or not guilty) of a lawsuit as it is not just about whether or not retail access will be approved but when it will happen. At time of writing, there are no external entities that give such predictions on a regular basis. Meanwhile experts who are informed and confident enough to give such assessments proved difficult to find. Even so, their opinions vary greatly.

Alternative Approaches

The investigation consisted of literature review, interviews with experts, pilot studies, understanding the deregulation process, and consultation with experts at conferences. This problem falls in the domain of judgmental forecasting and behavioral decision making. Should we use experts or non-experts? Should we use a judgmental or a non-judgmental (i.e. quantitative) method? If we elicit judgments, should it be individual unaggregated, mathematically aggregated, Delphi, or behaviorally aggregated (through a group meeting)? What is the easiest and best elicitation method: final probabilities, e.g. p(1); conditional probabilities, e.g. p(1|a); verbal assessment, e.g. "most likely"; or pairwise comparison, e.g. "p(1) > p(2)." If non-judgmental, should we use frequentist as opposed to subjective probabilities or build a statistical model? [See Kleindorfer et al (1993) p. 103 for a discussion of the three major schools of probability: classical, frequentist, and subjective.] The following figure illustrates my framework for investigating possible approaches.

An unaggregated individual assessment may be subject to bias, as described in the seminal work of Kahneman et al (1982) on heuristics and biases of individual judgment. Aggregation (mathematical or behavioral) of individual judgments produces somewhat better judgments with less variability if there is heterogeneity of opinion. A mathematical aggregation involves either simple or weighted averaging. Ferrell (1985) notes that an individual expert assessment is often better than an aggregation of non-experts, as ignorance cannot be averaged out. The method of mathematical aggregation of probabilistic judgments has strong predictable effects on calibration and accuracy. The Delphi method refers to an iterative method of obtaining an individual assessment, aggregating it with other assessments, feeding back the result to the individual to get a re-assessment until one assessment is achieved. Ferrell generalizes that Delphi refers to any procedure for assessing opinions that uses anonymous responses, some kind of feedback, and at least one iteration. Behavioral aggregation refers to group consensus through negotiation.

Non-judgmental methods involve using objective data such as whether or not an event has occurred, the duration of an event, the actual outcome of a decision, the number of pro-competition advocates, etc. Some combination of these statistics will give a frequentist probability distribution. There are also statistical modeling techniques that may be used to build a better predictor than individual indicators.

Table 2 summarises the main approaches investigated, with the first four methods pilot tested.

Group Scenario-Contingent Assessment

This approach requires a group of assessors to agree on how likely access will be achieved by a certain year. There are three scenarios: best, base, and poor. Best case refers to the situation where we are successful in influencing the process, the decision makers, and everything else goes in our favor. Base case reflects the status quo.

Probabilities are given for p(a), p(b), and p(c) where a = best case, b = base case, c = poor case. To get the probability that retail access will start in a particular year, regardless of which case (scenario), we must use Bayes’ Theorem and conditional probabilities, which have not been elicited. The probability that retail access starts in year: p(x) = p(x|a)p(a) + p(x|b)p(b) + p(x|c)p(c), where x = start of retail access in year 1, 2, or 3. The conditional probabilities p(x|a), p(x|b), and p(x|c) have not been defined. If p(1|a) = 1, p(2|b) = 1, and p(3|c) = 1, then p(1) = p(a), p(2) = p(b), and p(3) = p(c). See figure 2 for an illustration.

Decision Analysis

Decision analysis encompasses a range of decision theoretic techniques, including decision trees, influence diagrams, and multi-attribute utility analysis. Decision trees and influence diagrams are complementary structuring tools for decision analysis. Decision trees capture the chronological sequence of decision and uncertain events, while influence diagrams capture conditionality and dependence characteristics of events. Besides being more compact than decision trees, influence diagrams reveal probabilistic dependence and information flow. See Ku (1995) for further details. An illustrative influence diagram in figure 3 shows that retail access dates will not be announced unless critical issues such as stranded cost recovery are resolved.

This process of getting a probability distribution on timing of access involves 1) interviewing well-informed experts individually, 2) drawing an influence diagram to capture the main events, 3) drawing a corresponding decision tree, 4) eliciting conditional and marginal probabilities, and 5) calculating them to get the final probability distribution. Figure 4 gives one way of formulating the conditioning events, in this case, state and federal legislation only.

The decision analytic approach appeals to the causal thinker. However, it requires the assessor to think conditionally or a facilitator to do the elicitation. Some assessors find it difficult to give probabilistic estimates. Over time, new events will appear, and the decision tree will need to be restructured to include them. For example, at time of writing, the probability that federal legislation will be introduced is quite high, and it is expected that it will impact state legislation. However, an unforeseen event such as a major power outage or a competing bill may lower the probability of federal legislation, such that another critical event should be included in the tree instead. This is also a relatively time-consuming process as there are many possible configurations.

The above decision tree structure could be used for scenario analysis and serve as a means to validate the holistic assessments of the judgmental method. These decision trees will need to be modified for those states, in particular New York, which do not require state legislation for rule-making on retail access. The development of generic decision tree structures for different clusters of states will help simplify the structuring and elicitation effort.

Survey or Polling Method

The survey method is suitable for gathering input from a large number of anonymous respondents. In this case, respondents must be well-informed about the deregulation issues. Although there is no face to face elicitation required in sending out surveys, there is considerable effort in interpreting and collating the responses. Unless the questions are made explicitly clear, there is always a need to question the respondent’s assumptions. The accountability concern mentioned earlier is removed if the respondents are anonymous. Anonymity, however, prevents going back to the respondent for clarification. Due to the dynamic nature of restructuring events, it is necessary to get all the responses back quickly and simultaneously as well as to quickly assimilate the results before they get out of date. This timeliness requirement posed most troublesome as the respondents had no incentive to respond other than "doing a favor."

Event Tracking

Active monitoring of event-based state progress helps to validate the scenario-continent timing assessments, gauge the extent of state progress in the process, track the pace of deregulation, and anticipate next steps. It will also help guide the assignment of conditional probabilities in the judgmental method. Finally, it appears from experience so far that several issues require resolution before retail access can occur. Such "critical" events as stranded cost recovery, ISO formation, and divestiture (to allow a level playing field) should be included in state monitoring.

As there are overwhelmingly more states that are considering retail access than not, it is possible to see patterns and trends in state progress over time. If we treat deregulation as a process that all states have to go through, then the event tracking method is equivalent to setting the milestones and monitoring the critical path of a project. The process is defined by sequential "project" stages. The period of time each state spends in each stage varies but is usually known with some degree of certainty. By use of accelerating and delay factors, we can track whether certain "announced" dates will be reached or not. Similarly, by tracking how close to resolving the critical issues, we can track how close to meeting the target date. There are several hypothetical effects that have intuitive validity in the event tracking approach, as follows.

Many states in the advanced stages of restructuring have posted timelines of expected start date and duration of each activity. Some consultants believe there may be delays due to the sheer number of issues that most be resolved and the physical realities of getting systems and procedures ready.

A number of external independent agencies monitors restructuring activity. The National Regulatory Research Institute provides monitors the restructuring activities of each state. Their internet web site is http://kcgl1.eng.ohio-state.edu/~hoag/nrri.html. The bi-monthly LEAP Letter summarizes deregulation activity state by state, as well as tracks whether or not a bill has been proposed or adopted.

Another nonjudgmental method is to cluster the states by similar stages in the deregulation process and similar predictive characteristics, such as similar circumstances (e.g. high retail rates, liberal policies, presence of strong competition advocate), geographic proximity (adjacent state activities), historical bias, and proportion of Investor Owned Utilities (IOU), Cooperatives (Coops), Municipals (Muni’s). Some of these characteristics have been identified in the survey results discussed in an earlier section, but due to low response rate, they have not been used for clustering. An example of such clustering is the tier classification method of Regulatory Research Associates (RRA) as shown in table 3. Another deregulation monitoring source John, Hengerer & Esposito (JH&E) groups the states into four tiers representing gas and power deregulation progress. Tier one is unbundling; two is emerging states; three is potential states; four is monitor states..

A stakeholder is someone who has an interest in a decision, is affected by the decision, and can influence the decision making process and outcome, but does not make the decision his/herself. In economics, stakeholder theory concerns the redefinition of property rights by providing a voice in a firm’s decision making process to individuals or groups who are affected by a firm’s actions, but are not necessarily shareholders or otherwise associated with the firm by conventional means of ownership. Stakeholder analysis involves the identification of stakeholder types, identification of contentious issues, understanding of each stakeholder’s position on each issue and negotiation strategy, and analysis of interactions.

In electricity industry restructuring, participants represent a diverse set of interests. These stakeholders mainly fall into the following categories: legislatures, energy service companies, retail merchants associations, marketers, commercial business interests, utilities, ratepayer advocates, public utility commissions, environmental groups, low-income advocates, industrial user groups, governor’s offices, residential interest groups, and independent power producers.

By assessing the relative weight of influence, we effectively produce a decision analysis model using frequentist probabilities attached to a stakeholder’s position on an issue, i.e. outcome of a decision. The combination of a stakeholder’s position and relative weight of influence can be used to assess whether there will be acceleration or delay to meeting the date of approval and access. Aside from stake-holder influence and interaction, there are external events or random factors that may also affect the outcome. For example, it is well known that decision makers, e.g. Commissioners or Senators, do not always use all information available to make a decision, or act as one might expect. A better understanding of behavioral decision making may be useful in this respect.

Stakeholder analysis sets the stage for other kinds of economic analyses, such as game theory (Roth, 1985) and causal simulation (system dynamics). These types of analyses improve our understanding of interactions and what might happen.

The methods discussed so far rely on an absolute scale of timing in year of access. Another approach is to rank the states relative to each other with respect to progress. Progress can be defined by the tier classfication system of RRA described earlier or other event tracking milestones. Although we can similarly create key characteristics, such as critical events or time to resolution of certain issues, as criteria for relative ranking, they do not easily translate into probabilities.

Aside from cluster and factor analysis alluded in a previous section, there are other statistical methods that hold potential. Assuming that all states will go through the same process, we should be able to expect those factors that predict for a certain state also be used for other states. Limited methodological discussions with decision scientists indicated that survival modeling and Bayesian regression are possible approaches. However, these have not been investigated further in this context.

Originated in the biological and medical field, survival modeling refers to the use of survival data to predict survival time. Survival time is broadly defined as the time to the occurrence of a given event, e.g. death, or in our case, approval of retail access and start date of retail access. Survival data can include survival time, response to a given treatment, patient characteristics, or in our case, time taken in each stage of state progress towards deregulation, state legislature’s response to public utility commission’s recommendations, state’s tier group, and other state characteristics. The output of a survival model is a probability distribution.

Bayesian regression falls into a class of linear statistical models of the form y = f(x₁, x₂, ..., x_n). The dependent variable y is the time to direct access in years. There is a probability distribution around y. The independent variables are indicators, which the same group of experts have agreed to have strongest influence in predicting the time to access. Probability distributions are revised over time, so the model gets refined, i.e. through posterior updates of prior distributions. See Broemeling (1985) for further details.

The main objective of this study was to find a simple, systematic, defensible, and acceptable methodology for predicting the timing of deregulation (start date of retail access) for all fifty states. Unbiased assessment can be achieved by interviewing an objective observer, i.e. one who is not a stakeholder, and also balancing the views from representative sample of stakeholders involved in each state. After pilot testing four out of seven possible approaches, we recommended a parallel approach of using both judgmental and non-judgmental methods to get monthly assessments. The judgmental method takes scenario-contingent expert assessments from those participants actively involved at the state and federal levels. The non-judgmental method refers to monitoring state progress by use of internal indicators and external monitoring entities whose results are published regularly. Due to the timeliness of the results, it is necessary to ensure the monitoring sources are in sync with each other as well as with the expert assessments.

The difficulties encountered in most of the approaches are not insurmountable as shown in table 4. They can largely be overcome by management directive and sufficient resources. Alternatively, they may be removed altogether if the exercise is conducted by an external entity who not only monitors state progress but can also ensure full response rate from a regular (repeatable) survey of a representative sample of stakeholders in each state and collated and updated in a timely manner.

As the pace of restructuring accelerates and the learning curve flattens, we can expect to find more experts. Existing experts will be able to see patterns in state progress to predict timing more accurately and with greater confidence. Similarly there may be more entities reporting and monitoring state progress, providing further "validation" to the assessments.

Figures and Tables

Table 1 Timing Certainty (illustrative probabilities that the year will be known, discussed, or will occur as said)

Figure 1 Possible Approaches

Table 2 Comparison of Alternative Approaches

Figure 2 Conditional Probabilities

Figure 3 Influence Diagram

Figure 4 Decision Tree

Figure 5 Probability Distribution

Table 3 RRA’s Classification

Table 4 Difficulties and Suggestions to Overcome Them

Author's Notes

References