Transcription of Push Data Science in Spark with sparklyr
{{id}} {{{paragraph}}}
Example: bankruptcy
Push Data Science in Spark with sparklyr
FdYARNM esosor fdfdfdfdfd 512m 120s 1g 1library( sparklyr ); library(dplyr); library(ggplot2); library(tidyr); (100) spark_install(" ") sc <- spark_connect(master = "local") import_iris <- copy_to(sc, iris, "spark_iris", overwrite = TRUE) partition_iris <- sdf_partition( import_iris,training= , testing= ) sdf_register(partition_iris, c("spark_iris_training","spark_iris_test")) tidy_iris <- tbl(sc,"spark_iris_training") %>% select(Species, Petal_Length, Petal_Width) model_iris <- tidy_iris %>% ml_decision_tree(response="Species", features=c("Petal_Length","Petal_Width") ) test_iris <- tbl(sc,"spark_iris_test") pred_iris <- sdf_predict( model_iris, test_iris) %>% collect pred_iris %>% inner_join( (prediction=0:2, lab=model_iris$ $labels)) %>% ggplot(aes(Petal_Length, Petal_Width, col=lab)) + geom_point() spark_disconnect(sc)Partition dataInstall Spark locallyConnect to local versionCopy data to Spark memoryCreate a hive metadata for each partitionBring data back into R memory for plottingA brief example of a data analysis using Apache Spark , R and sparklyr in local modeSpark ML Decision Tree ModelCreate reference to Spark tableDisconnect Collect data into R Share plots, documents, and apps Spark MLlib H2O ExtensionCollect data into R for plottingTransformer function dplyr verb Direct Spark SQL (DBI) SDF function (Scala API) Export an R DataFrame Read a file Read existing Hive tableData Science i
ft_imputer() - Imputation estimator for completing missing values, uses the mean or the median of the columns ft_index_to_string() - Index labels back to label as strings ft_interaction() - Takes in Double and Vector type columns and outputs a flattened vector of their feature interactions Translates into Spark SQL statements DPLYR VERBS Wrangle
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: