Announcements

• April 9, 2011 (SAT):
  Released the TAU 2011 contest presentation with results. Please look under the Contest Benchmarks and Results section.
• March 1, 2011 (SUN):
  We start to accept the final submission from March 7, to make sure you get all problems fixed from alpha code. The final deadline is extended to March 11.
  All teams should have received the first round of feedback for the alpha code.
  Update the Contest Benchmark Section.
  Update the Submission Information Section.
  Update the Evaluation and Ranking Section.
  Update the Q&A Section.
  
• Jan 25, 2011 (TUE): Released important submission guidelines including contest machine configuration and evaluation.
• Jan 12, 2011 (THUR): We have sent out the contest notification email to all teams. If you have not received an email with title "Tau 2011 Contest Notification", please let us know.
• Jan 12, 2011 (WED): New contest website is up with instructions.
  Please read the website "Benchmark_Format" under Benchmark Format. This has important information regarding the benchmark format.
  

Contest Benchmarks and Results

1. Test benchmarks: ibmpg1 to ibmpg6 from this site. Two new benchmarks will be added soon (only these two benchmarks will be released before contest).
2. Five new benchmarks will be released soon.
3. TAU 2011 Contest Presentation with Results.

Contact Information

• This year the contest will be administered by Raju Balasubramanian, Zhuo Li and Frank Liu from IBM. Please send all emails regarding the contest to taucontest2011@gmail.com.
• Please add "TAU2011_Contest" to your subject line.

Schedule

Submission Information

1. Please name your binary as your solver name. We do not want to have 5 "solver" and 5 "main" binaries.
2. You do not need to print extra information if that takes runtime. All we need is the solution file and we will count the runtime ourself.
3. If you have any special performance tuning parms, please remove them or hardcode inside your program. For the contest, we only take two options, one is for input file and one is for output file.
4. Please provide an arg to your program so that I can specify the solution file name.
5. Please try to submit a statically-linked binary as it helps with portability. (use gcc -static).
6. Each team is allowed to submit a single binary that should run on all the benchmarks.
7. For this contest we would like to evaluate single-Core versions of the power grid simulation algorithms. Hence, no parallel implementations are allowed.
8. No precomputed information can be used to influence the current run. The run directories will be cleaned prior to each run.
9. Purpose of alpha (preliminary) binary submission:
   The purpose of the alpha binary submission is: (a) to verify that I can run your binary on the contest machine. (b) to test that your binary is able to produce required formats on the test benchmarks.
10. Memory limit per job: 64GB RAM, 16GB swap memory
11. Machine configuration for the contest:
    • OS: Ubuntu 10.10. Linux 2.6.35-22-server
    • CPU: 64-bit Intel(R) Xeon(R) CPU E7210 @ 2.40GHz
    • GCC version: gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5)
      Target: x86_64-linux-gnu
    • BLAS: libblas-dev, 1.2-7. liblapack-dev, 3.2.1-8
    • Tcl: 8.5.8-2build1
    • Information from /proc/cpuinfo:

          processor       : 0
          vendor_id       : GenuineIntel
          cpu family      : 6
          model           : 15
          model name      : Intel(R) Xeon(R) CPU           E7210  @ 2.40GHz
          stepping        : 11
          cpu MHz         : 2398.625
          cache size      : 4096 KB
          physical id     : 3
          siblings        : 2
          core id         : 0
          cpu cores       : 2
          apicid	    : 12
          initial apicid  : 12
          fpu             : yes
          fpu_exception   : yes
          cpuid level     : 10
          wp              : yes
          flags	    : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca lahf_lm tpr_shadow vnmi flexpriority
          bogomips        : 4797.25
          clflush size    : 64
          cache_alignment : 64
          address sizes   : 40 bits physical, 48 bits virtual
          power management:
        

Evaluation and Ranking

• The metrics for evaluation will be based on four categories
  • CPU;
  • Memory;
  • Maximum absolute error of the voltages among all nodes.
  • Average absolute error of the voltages among all nodes.
  
  For each team, their power grid simulator will be run on all benchmarks. For each benchmark, a score is given in each category according to the simulation results, and the sum of the scores of all categories is the score for this benchmark. The final score of each team is the weighted sum of the scores of all benchmarks, where the weight is computed based on the benchmark size. Here are some additional information for the score function in each category.
  • CPU: The CPU score is from 0 to 100. It will be computed by normalizing the runtime of the the submitted simulator to the runtime of "an internal direct solver" implemented in matlab. If the runtime is longer than the internal solver, the score will be 100.
  • Memory: The Memory score is from 0 to 100. It will be computed by normalizing the memory of the the submitted simulator to the memory of "an internal direct solver" implemented in matlab. If the memory is bigger than the internal solver, the score will be 100.
  • Maximum absolute error: The score is from 0 to 100. If the maximum absolute error is less than 1 mV, the score is 0. If the max absolute error is larger than 10 mV, the score is 100.
  • Average absolute error: The score is from 0 to 100. If the average absolute error is less than 0.1 mV, the score is 0. If the average absolute error is larger than 1 mV, the score is 100.
  • If your simulator runs out of memory for a benchmark, it gets 400 score for this benchmark automatically.
  • If your simulator runs more than 10x of our matlab direct solver for a benchmark, it gets 400 score for this benchmark automatically even though the simulator may get exact solution.
  • We will have an internal program to detect if the solution is completely out of range for a benchmark (i.e., from random number generation). If it happens, then the simulator will get 400 score for this benchmark even though the CPU and memory score may be 0.

Utility Scripts

• None.

Frequently Asked Questions

1. Some of you had some questions about one of the ranking rules, specifically the rule stating that
   
   "if your simulator runs more than 10x of our matlab direct solver for a benchmark, it gets 400 score for this benchmark automatically even though the simulator may get exact solution".
   
   In order to have a basis for comparison, we knew that we needed a baseline to allow us to weigh the various performance metrics of each of your codes. As you know, the metrics we chose were CPU time, Memory, Average Error, and Maximum Error. Since each of these quantities has a different unit, we needed to normalize them to the same basis. To do this, we decided that it would not be fair to use a mature power grid analysis tool (such as the one IBM uses internally), so instead we implemented a simple and short set of small scripts to read the Spice netlists, create the corresponding matrix, and solve that matrix using Matlab.
   
   We believe that such a simple approach represents the "zero effort" and "zero error" baseline for how one might solve a power grid. We intend to put the scripts on the web site after the contest is finished. But in addition we firmly believe that the contestant should strive to do much better than such a simple minded approach, hence we adjusted the scoring function so that if the competing simulator runs more than ten times slower, we need to indicate that the approach is clearly impractical.
   If any of you have more questions about this, please feel free to email us.
   
   
2. Is the simulator going to be run on a single-core CPU, a multi-core CPU, or GPU?
   For this yeart contest we would like to evaluate single-threaded versions of the simulation algorithms. The hardware acceleration part will not be considered. Each team is required to submit the binary code which can be run on a single core Linux machine. For the future contest (if there will be one), we may provide more advanced hardware platforms (i.e., GPU).


3. Will the contest fix the simulation method that is going to be used, or we can choose any feasible methods, like Cholesky, multi-grid, and CG?
   No.


4. Do we have to write the functions for the basic matrix computations, like Cholesky decomposition?
   It is the team's choice. If you use external library, you need to provide static code and make sure the external library can run on the platform we release.


5. Do we have to write the parser to read the netlist for our simulator?
   Yes.


6. Will the benchmark have RC information and do we need to solve transient cases?
   No. We only only focus on DC solution this year.

Benchmark Format

Participating Teams

• Coming soon.

Call for Participation