hadoop - how to run a mapreduce job on amazon's elastic mapreduce (emr) cluster from windows? -


i'm trying learn how run java map/reduce (m/r) job on amazon's emr. documentation following here http://aws.amazon.com/articles/3938. on windows 7 computer.

when try run command, shown information.

./elasticmapreduce-client.rb runjobflow streaming_jobflow.json  

of course, since on windows machine, type in command. not sure why, particular command, there not windows version (all commands shown in pairs, 1 *nix , 1 windows).

 ruby elastic-mapreduce runjobflow my_job.json 

my question how submit/run job windows amazon's emr using command line interface (on windows)? i've tried searching online, taken wild places. appreciated.

thanks.

hmmm. i'm not sure how old example runjobflow is... i'd ignore it.

are able run?

localhost$ elastic-mapreduce --describe 

once can should play directly on cluster shake out exact steps need do... it's worth doing don't have start/stop cluster bazillion times.

localhost$ elastic-mapreduce --create --alive --num-instances 1 localhost$ elastic-mapreduce -j j-your_id_here --ssh  cluster$ hadoop jar my.jar -d some=1 -d args=1 blah blah cluster$ hadoop jar some_other_jar.jar -d foo -d bar cluster$ ^d  localhost$ elastic-mapreduce -j j-your_id_here --terminate 

then when you're happy steps , need have run headless (say, cron) can have emr orchestrate steps (including cluster self terminating @ end)

localhost$ elastic-mapreduce --create --num-instances 1 localhost$ elastic-mapreduce --jar my_jar.jar --args "-d,some=1,-d,args=1,blah,blah" localhost$ elastic-mapreduce --jar some_other_jar.jar --args "-d,foo,-d,bar" 

i'd explore --json stuff if need more complex steps, it's bit cryptic , hard right first time...


Comments

Popular posts from this blog

delphi - How to convert bitmaps to video? -

jasper reports - Fixed header in Excel using JasperReports -

python - ('The SQL contains 0 parameter markers, but 50 parameters were supplied', 'HY000') or TypeError: 'tuple' object is not callable -