setup local development environment for pyspark in Windows

Install Java(8+)
https://www.java.com/en/download/help/download_options.xml#windows
https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Add to Environment Variables
JAVA_HOME C:\Programs Files\jdk1.8.0_192
add to Path C:\Program Files\Java\jdk1.8.0_191\bin
C:\Program Files\Java\jre1.8.0_191\bin

1. Setup Hadoop winutils:
   Download from this GitHub repo https://github.com/steveloughran/winutils
   Copy bin folder under Hadoop 2.7.1 to your location. For example C:\ProgramData\wintuils
   Add to Environment Variables
   HADOOP_HOME C:\ProgramData\winutils
   add to Path %HADOOP_HOME%\bin

2. Install Anaconda(5.3)
   https://www.anaconda.com/download/
   Downgrade python to 3.6.5 since python 3.7 may not compatible with some packages

3. Install Spark(2.3.2 recommended) with Hadoop 2.7
   Download .tgz from https://www.apache.org/dyn/closer.lua/spark/spark-2.3.2/spark-2.3.2-bin-hadoop2.7.tgz

You might need to install 7zip tp unzip .tgz, move unzipped folder to a separate location, for example, C:\

Add to Environment Variables
SPARK_HOME C:\spark-2.3.2-bin-hadoop2.7
Add to Path %SPARK_HOME%\bin
To verify it, open Command Prompt, cd to bin folder of SPARK_HOME, and type pyspark

1. Install Pycharm
   https://www.jetbrains.com/pycharm/download/#section=windows
   set up interpreter in pycharm, using Conda Environment and adding packages by click "+"

2. For command line run, using pip install -r requirements.txt
   Using pip freeze > requirements.txt when updating dependencies
   Or open Pycharm the Settings/Preferences dialog (Ctrl+Alt+S) and select Tools | Python Integrated Tools.
   In the Package requirements file field, type the name of the requirements file or click the browse button and locate the desired file.

Click OK to save the changes.