hadoop - how to run a mapreduce job on amazon's elastic mapreduce (emr) cluster from windows? -
i'm trying learn how run java map/reduce (m/r) job on amazon's emr. documentation following here http://aws.amazon.com/articles/3938. on windows 7 computer.
when try run command, shown information.
./elasticmapreduce-client.rb runjobflow streaming_jobflow.json
of course, since on windows machine, type in command. not sure why, particular command, there not windows version (all commands shown in pairs, 1 *nix , 1 windows).
ruby elastic-mapreduce runjobflow my_job.json
my question how submit/run job windows amazon's emr using command line interface (on windows)? i've tried searching online, taken wild places. appreciated.
thanks.
hmmm. i'm not sure how old example runjobflow is... i'd ignore it.
are able run?
localhost$ elastic-mapreduce --describe
once can should play directly on cluster shake out exact steps need do... it's worth doing don't have start/stop cluster bazillion times.
localhost$ elastic-mapreduce --create --alive --num-instances 1 localhost$ elastic-mapreduce -j j-your_id_here --ssh cluster$ hadoop jar my.jar -d some=1 -d args=1 blah blah cluster$ hadoop jar some_other_jar.jar -d foo -d bar cluster$ ^d localhost$ elastic-mapreduce -j j-your_id_here --terminate
then when you're happy steps , need have run headless (say, cron) can have emr orchestrate steps (including cluster self terminating @ end)
localhost$ elastic-mapreduce --create --num-instances 1 localhost$ elastic-mapreduce --jar my_jar.jar --args "-d,some=1,-d,args=1,blah,blah" localhost$ elastic-mapreduce --jar some_other_jar.jar --args "-d,foo,-d,bar"
i'd explore --json stuff if need more complex steps, it's bit cryptic , hard right first time...
Comments
Post a Comment