Set USE_JOYSTICK_AS_DEFAULT to False in cfg_complete.py

Improvements in tub and training to manage new datastore and new tub GUI

Fix tflite input/output conversion.

Add performance monitor

This is the default in v3. Setting this to `True` by default will not activate the web controller and caused some confusion to users.

myconfig.py default scene name fix

Add back training history plot

Patch to #752 - fix wrong operator in pip.

Add missing `DEFAULT_AI_FRAMEWORK`. `donkey train` won't run without this.

* Rename ubuntu.yml to pc.yml and update docs accordingly.

* Added missing update in travis.

Fixed FindCar function for RPi4

* Renamed function for session id creation in manifest and stored dictionary of last_id, last_full_id and all_full_ids in the metadata so no parsing needs to happen. Added more comment to the manifest to explain the session id logic. Added `if` around session_id update flag, so flag gets set only once.
* Reverted tub test and created new tub writer test instead
* Removed shutdown hook from tub writer as it will be closed by the vehicle
* When session_id data gets inserted into the metadata, only modify the corresponding line (actually the seekable also modifies the lines below, there is still room for improvement in that method)
* Minor performance upgrades and improved timing in fast forward loop in UI

* Fixed manifest iterator to avoid too deep recursive function calls which segfaults when deleting many records
* Automatic addition of '_session_id' field in the Tub. This mimics the naming convention we used to have for old-style tubs before: YY-MM-DD_N, where N is a running integer. All session ids of a tub are stored in the manifest.json file. The tub needs to receive records and needs to get closed for this entry to be updated.
* Added test for the new tub and tub writer logic

Tub GUI development:
* Support starting from 'donkey ui' command
* All relevant settings are stored in ~/.donkeyrc file which is editable

Training improvements:
* You can now put a predicate for tub records into the myconfig.py to filter records for training. The config variable TRAIN_FILTER has to be set to that function.
* Improve training performance by using tf prefetch
* Default AI framework to 'tensorflow' in training

Others:
* Update docs
* Update patch version number to 4.2.0
* Reduced run time of test_train using only 5 epochs and adjusting thresholds accordingly
* Reduced run time of torch tests using 3 epochs - these were very slow (like 25+ m on MacBook)
* Minor updates in docstrings and switching to f-strings
* Fixed duplicated function in keras.py

Co-authored-by: Chris Anderson <gca2@earthlink.net>

Co-authored-by: Chris Anderson <gca2@earthlink.net>

* - Update tensorflow version to 1.15.0
- Add doc for Jetpack 4.4
- Remove opencv installation

* Add dataset and pre-trained model page

* fix issue #634 (#646)

- remove jquery dependency
- add clearTimeout

* Allow overriding WEB_CONTROL_PORT from env variable

* pin h5py version

* fix h5py version for mac

* * Update doc with donkey train command. (#709)

* Update doc with developer section for building own models in donkey 4.1
* Integrate changes from PR feedback

Co-authored-by: Jonathan Tse <jonathan@tsez.net>
Co-authored-by: Tawn Kramer <tawnkramer@gmail.com>
Co-authored-by: DocGarbanzo <47540921+DocGarbanzo@users.noreply.github.com>

* Added in PyTorch and PyTorch Lightning to train a DC model

Successfully able to train a ResNet18-based model using PyTorch
Lightning.

* Removed hard-coded max number of epochs (used for debugging)

* Added an inference transform to ResNet18 to convert PIL -> tensor

* Unsqueezed input tensor during inference for batch dimension

* Reshaped ResNet output from (1, 2) to (2,)

* Added the ability to resume training from a checkpoint

* Added helper print message when tensorboard logging is enabled

* Updated docopt arguments for train.py. Made checkpoint optional

* Changed TorchTubDataset from sub-classing Dataset to IterableDataset

This was done in response to https://github.com/autorope/donkeycar/pull/706#discussion_r548137252

* Renamed load_image_arr to load_image. Updated load_pil_image

load_pil_image will now handle converting the image to greyscale
(vs. this being done in load_image).

* Updated enviroments for Mac and Ubuntu. Set Python=3.7

* Updated installation documentation. Added script to setup Nano

Updated the installation instructions for Ubuntu, Mac, and Windows.
Clarified a common issue that occurs when running pip install -e .[pc]
with ZSH.

Also added a script to setup the Jetson Nano and updated the documentation
for the Nano (it previously was installing tensorflow 1.x).

* Added torch flag to setup.py to install pytorch

* Moved pytorch training into base.py and removed from train.py

* Moved Jetson Nano python package installation into requirements.txt

* Formatted with PEP8 to clean up pytorch code

* Updated docs to provide work-around for ZSH pip install -e .[pc]

* Removed duplicate dependencies in conda env files

* ResNet18 torch model now returns training loss history

* Added test file for PyTorch training

Still need to make sure this passes Travis CI.

* Added lightning_logs to .gitignore

* You can now specify the default AI framework to use in config.py

This reduces the number of command line arguments you are required
to provide.

* get_model_by_type for PyTorch now lazy imports ResNet18

* Added help message to torch_train. Got rid of linear model type

* Updated pytorch tests and fixed some syntax errors

* ResNet18 example input shape updated to be (B, 3, 224, 224)

Also now passing output_shape to load_resnet18 to modify how many
output classes are used

* No longer pinning requirement versions for Jetson Nano

* Fixed formatting in setup.py

There is a small typo in docs/guide/host_pc/setup_windows.md.

Should read `unfamiliar` rather than `unfarmilar`.

* Minor changes for 4.1
* Update conversion script to translate discontinuous data.
* In developer guide add disclaimer for version and correct intra-page links.

* * Update doc with donkey train command.
* Update doc with developer section for building own models in donkey 4.1
* Integrate changes from PR feedback

* Incorporate PR feedback
* Add empty records concept to tub
* Minor updates to conversion script

* Incorporate PR feedback
* Add empty record type into conversion script

* Add support for multi-dimensional model input and making the x, y interface symmetrical on the model / training interface: (#707)

- x/y_transform extract x, y as numpy arrays or floats out of the record
  - x/y_translate convert the numpy arrays of floats into tf-readable dictionaries used in tf data.
* Simplify model interface by implementing output_types() directly in the base class using output_shapes() dictionary.
* Adding developer guide for own model development
* Updated donkey command documentation
* Improve asserts and type hints in keras.py
* Added missing __init__.py in parts module.
* Add cool ascii text for donkey init and update yml and setup files including mypy
* Remove model training test from Travis and change the test to relative convergence. This avoids random fall overs in CI.
* Added test of tf.data as used in the training pipeline through re-implementation of data transformation from tub records to tf expected dictionaries, for all currently supported models.

* Improve pipeline use: move from building list of pipelines of single transforms to building a single pipeline with a list of transforms (actually just looping through function to go from TubRecord -> image -> augment -> normalise -> x and TubRecord -> y).

Fixed TfmIterators and TfmIterables.
* Iterables are the containers and are sized - these are the user objects
* Iterators are protocol objects to allow iteration, they have no logic and are local to the Iterables
* build/map_pipeline both return sized Iterables
* removed all batch logic, this is not required
* still commented but left code that uses generator based pipeline as this is simpler code

Using new small temporary pipeline generator
* this keeps the TubSequence lazy and avoids to roll out the pipeline into a list
* added a test to check consistency of the pipeline
* remove empty (after moved) augmentation file
* removed augmentation from old tub (as it's not needed and we removed the old augmentation)

New pipeline changes:
* moved augmentation into own class that is used above and can be used a  threaded or non-threaded part
* moved train functionality out of template and added 'donkey train', train.py just a simple dummy script for backward compatibility

* Address code reviews:
* Re-base on current dev to use un-altered sequence.py
* Add iterator consistency test to pipeline tests
* Undo changes in fast_stretch.py
* better tf shape manipulation
* small code improvements in training.py
* remove sleep in augment part

* Address code reviews:
* Add clearing of tubrecord list and minor renamings

change the public mqtt server name to one that's listening

The telemetry mqtt test is breaking w/ a timeout on the connection to the server

* Implement a Lazy transformable pipeline.
* Implement basic batching. However, this will need to be improved
  further for models with multiple outputs.
* Replace the old `Sequence` implementation used with a new `Pipeline`.

Test: Ran end to end tests.

* This makes reading a 100x faster.