Why is there a limit on the number of counters allowed in a Hadoop map reduce job?
I am using Hadoop map-reduce and got a CountersExceededException when I tried to programatically create many job counters. I know that I can increase the number of counters that are allowed through the configuration files, but does anyone know:
a) why there is a limit on map-reduce counters
b) if it is a good/bad idea to increase the max number of map-reduce counters?
I am using hadoop 0.20.2
See the section on Counters in this post.
Counters represent global counters, defined either by the Map/Reduce framework or applications. Applications can define arbitrary Counters and update them in the map and/or reduce methods. These counters are then globally aggregated by the framework.
Counters are appropriate for tracking few, important, global bits of information. They are definitely not meant to aggregate very fine-grained statistics of applications. Counters are very expensive since the JobTracker has to maintain every counter of every map/reduce task for the entire duration of the application.
Just in case you want to up the maximum number of counters.
Last I saw the code, the maximum number of counters has been hard coded in the code and not configurable. Please check the blog entry I have written some time back for more details and also go through the latest code to check if this has been modified to make it configurable.