Updates and Fixes to Make spark-memory work on opensource spark

[AndrewKL]

Updates and Fixes to Make spark-memory work on opensource spark

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ spark-memory-core_2.11 ---
[INFO] Building jar: /Users/loandrew/Workspaces/Spark-memory/src/Spark-memory/core/target/spark-memory-core_2.11-0.1.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (make-assembly) @ spark-memory-core_2.11 ---
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] scala/ already added, skipping
[INFO] scala/util/ already added, skipping
[INFO] Building jar: /Users/loandrew/Workspaces/Spark-memory/src/Spark-memory/core/target/spark-memory-core_2.11-0.1.0-SNAPSHOT-jar-with-dependencies.jar
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] scala/ already added, skipping
[INFO] scala/util/ already added, skipping
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for spark-memory-parent_2.11 0.1.0-SNAPSHOT:
[INFO] 
[INFO] spark-memory-parent_2.11 ........................... SUCCESS [ 14.110 s]
[INFO] spark-memory-core_2.11 ............................. SUCCESS [ 22.685 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  36.972 s
[INFO] Finished at: 2019-08-21T16:37:43-07:00
[INFO] ------------------------------------------------------------------------

[squito]

Thanks for doing this -- yes it totally makes sense, this was last updated before the api was finalized.

Other than removing the extra file, this looks good, thanks!

[squito]

can you remove this file from the pr please?

[squito]

Also btw, a lot of the functionality of this plugin was recently merged to master in apache spark (SPARK-26329 / apache/spark#23767). Its not exactly the same -- that makes it easier to capture metrics at a faster rate and still include something reasonable in the event logs. But the major problem with that approach is we don't get any info from right before the executor gets killed suddenly.