Everything was working nicely for 2 hours. Then, I started getting: WARNING - neptune.internal.channels.channels_values_sender - Failed to send channel value: Server error. Please try again later. and later a number of following stacktraces:

Exception in thread Thread-26:                                                                                            
Traceback (most recent call last):                                                                                            
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917, in _bootstrap_inner                   
    self.run()                                                                                                                         
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 121, in run
    self._process_batch()                                                                                             
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 133, in _process_batch
    self._send_values(self._values_batch)                                 
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 158, in _send_values
    self._experiment._send_channels_values(channels_with_values)                                                                       
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1080, in _send_channels_values
    self._backend.send_channels_values(self, channels_with_values)
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188, in wrapper               
    return func(*args, **kwargs)
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 513, in send_channels_values
    channelsValues=input_channels_values                                                         
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 254, in __call__                       
    request_config=request_config,                                                 
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 193, in request                           
    self.authenticated_request(sanitized_params),
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 234, in authenticated_request
    return self.apply_authentication(requests.Request(**request_params))                                                                                  
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 239, in apply_authentication
    return self.authenticator.apply(request)                                                                                                                 
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/oauth.py", line 87, in apply
    self.auth.refresh_token_if_needed()                                   
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188, in wrapper
    return func(*args, **kwargs)
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/oauth.py", line 51, in refresh_token_if_needed
    self._refresh_token()
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/oauth.py", line 54, in _refresh_token
    self.session.refresh_token(self.session.auto_refresh_url)
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests_oauthlib/oauth2_session.py", line 374, in refresh_token
    self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 421, in parse_request_body_response
    self.token = parse_token_response(body, scope=scope)
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 431, in parse_token_response
    validate_token_parameters(params)
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 441, in validate_token_parameters
    raise MissingTokenError(description="Missing access token parameter.")
oauthlib.oauth2.rfc6749.errors.MissingTokenError: (missing_token) Missing access token parameter.

Added function that allows users to create parallel coordinates plot.

Few use cases:

1. Visualize interactive plot in Notebook
2. Save visual as standalone html

Regarding plotting:

1. compare multiple experiments
2. inspect experiments lineage

Im trying to use hiplot with neptune. I get the following exception:

Traceback (most recent call last):
  File "c:/Users/ali/Desktop/pythonProjects/Deep-learning-algo-master/src/TestDriver.py", line 204, in <module>
    make_parallel_coordinates_plot(html_file_path='my_visual.html',
  File "C:\Users\ali\miniconda3\envs\iptoki\lib\site-packages\neptunecontrib\viz\parallel_coordinates_plot.py", line 141, in make_parallel_coordinates_plot
    assert df.shape[0] != 0, 'No experiments to show. Try other filters.'
AssertionError: No experiments to show. Try other filters.

in my code: I call the make_parallel_coordinates_plot after the optimization step. So in Neptune, I can see for example, in my workspace, the number of the experiment let say ali-22. I tried to give the experiment id explicitly by setting the experiment_id = 'ali-22' in make_parallel_coordinates_plot but the same exception is thrown. Any suggestion to solve this is so appreciated.

Integration with XGBoost.

What it does?

• logs metrics during training
• at the end of training it logs model, importances, visualized trees -> you can switch off al of them if you wish
• docstrings ready

More info

• it works with train, cv and sklearn API form XGBoost
• in the case of CV we log model, importances and trees for each folds' booster -> it can be really cool to the users.

@jakubczakon please wait until all tests are green :)

i'm trying on my pycharm with conda envs, this lightGBM case on neptune.ai, it throw a series of errors, great software, plz upgrade

F:\Anaconda\envs\kaggle\python.exe "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\pydevconsole.py" --mode=client --port=3967
import sys; print('Python %s on %s' % (sys.version, sys.platform))
sys.path.extend(['F:\\iPynb\\LendingClub', 'F:/iPynb/LendingClub'])
Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.18.1
Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)] on win32
In[2]: runfile('F:/iPynb/LendingClub/main.py', wdir='F:/iPynb/LendingClub')
Traceback (most recent call last):
  File "F:\Anaconda\envs\kaggle\lib\site-packages\IPython\core\interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-512b071ef48b>", line 1, in <module>
    runfile('F:/iPynb/LendingClub/main.py', wdir='F:/iPynb/LendingClub')
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "F:/iPynb/LendingClub/main.py", line 14, in <module>
    from scikitplot.metrics import plot_roc, plot_confusion_matrix, plot_precision_recall
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scikitplot\__init__.py", line 2, in <module>
    from . import metrics, cluster, decomposition, estimators
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scikitplot\metrics.py", line 14, in <module>
    from sklearn.metrics import confusion_matrix
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\__init__.py", line 80, in <module>
    from .base import clone
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\base.py", line 21, in <module>
    from .utils import _IS_32BIT
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\__init__.py", line 23, in <module>
    from .class_weight import compute_class_weight, compute_sample_weight
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\class_weight.py", line 7, in <module>
    from .validation import _deprecate_positional_args
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\validation.py", line 25, in <module>
    from .fixes import _object_dtype_isnan, parse_version
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\sklearn\utils\fixes.py", line 18, in <module>
    import scipy.stats
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\stats\__init__.py", line 388, in <module>
    from .stats import *
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\stats\stats.py", line 174, in <module>
    from scipy.spatial.distance import cdist
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\spatial\__init__.py", line 102, in <module>
    from ._geometric_slerp import geometric_slerp
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\spatial\_geometric_slerp.py", line 8, in <module>
    from scipy.spatial.distance import euclidean
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\spatial\distance.py", line 121, in <module>
    from ..special import rel_entr
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\special\__init__.py", line 645, in <module>
    from ._ellip_harm import (
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "F:\Anaconda\envs\kaggle\lib\site-packages\scipy\special\_ellip_harm.py", line 4, in <module>
    from ._ellip_harm_2 import _ellipsoid, _ellipsoid_norm
  File "F:\IDE\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "_ellip_harm_2.pyx", line 9, in init scipy.special._ellip_harm_2
ModuleNotFoundError: No module named 'scipy._lib._ccallback'```

When using sacred the result variable can be either a float (which NeptuneObserver code currently supports) or multiple results (a tuple, which causes a crash do to neptune.log_metrics() not supporting tuples).

This is a quick fix I made ... but it would be nice that if one of the elements of the tuple is object (e.g. If parts of my results is a buffer / dataset) I could also store it

Hello,

I started to use log_binary_classification_metrics, which is quite handy. I've incorporated it into an evaluation pipeline, which is run every N steps while training. That is fine as for the numeric variables this method logs to neptune. However the super-handy metric_charts are all added to a single big 'bucket' in neptune, and I cannot understand which charts are from which epoch/step. see e.g. here I have logged this twice and I have two confusion matrices, etc...:

I could use the 'prefix' parameter to log_binary_classification_metrics, and store epoch/step numer there - but this makes my metrics messy, as every numeric parameter at every step start to have different name. what I suggest is either:

1. one can pass additional parameter like 'title', which would be included in the title of every generated image
2. make 'prefix' work only for charts?

you may have other ideas on how to solve this problem easily, I hope I made it understandable

Given code:

y_true = np.array([1, 1, 0, 0])
y_pred = np.array([[0.00519159, 0.9948084 ],
        [0.00550756, 0.99449253],
        [0.00536389, 0.9946362 ],
        [0.00568774, 0.99431235]])

log_binary_classification_metrics(y_true, y_pred, threshold=0.5)

I get:

IndexError                                Traceback (most recent call last)
<ipython-input-24-9c6f1c89bf03> in <module>
----> 1 log_binary_classification_metrics(y_true, y_pred, threshold=0.5)

...site-packages/neptunecontrib/monitoring/metrics.py in log_binary_classification_metrics(y_true, y_pred, threshold, experiment, prefix)
     82     log_classification_report(y_true, y_pred[:, 1] > threshold, experiment=_exp, prefix=prefix)
     83     log_class_metrics(y_true, y_pred[:, 1] > threshold, experiment=_exp, prefix=prefix)
---> 84     log_class_metrics_by_threshold(y_true, y_pred[:, 1], experiment=_exp, prefix=prefix)
     85     log_roc_auc(y_true, y_pred, experiment=_exp, prefix=prefix)
     86     log_precision_recall_auc(y_true, y_pred, experiment=_exp, prefix=prefix)

...site-packages/neptunecontrib/monitoring/metrics.py in log_class_metrics_by_threshold(y_true, y_pred_pos, experiment, channel_name, prefix)
    284     _exp = experiment if experiment else neptune
    285 
--> 286     figs = _plot_class_metrics_by_threshold(y_true, y_pred_pos)
    287 
    288     for fig in figs:

...site-packages/neptunecontrib/monitoring/metrics.py in _plot_class_metrics_by_threshold(y_true, y_pred_positive)
    711             continue
    712         else:
--> 713             best_thres, best_score = _get_best_thres(scores_by_thres, name)
    714             fig, ax = plt.subplots()
    715             ax.plot(scores_by_thres['threshold'], scores_by_thres[name])

...site-packages/neptunecontrib/monitoring/metrics.py in _get_best_thres(scores_by_thres, name)
    806         best_res = scores_by_thres[scores_by_thres[name] == scores_by_thres[name].max()][['threshold', name]]
    807     position = len(best_res) // 2
--> 808     result = best_res.iloc[position].to_dict()
    809     return result['threshold'], result[name]

...site-packages/pandas/core/indexing.py in __getitem__(self, key)
   1766 
   1767             maybe_callable = com.apply_if_callable(key, self.obj)
-> 1768             return self._getitem_axis(maybe_callable, axis=axis)
   1769 
   1770     def _is_scalar_access(self, key: Tuple):

...site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   2136 
   2137             # validate the location
-> 2138             self._validate_integer(key, axis)
   2139 
   2140             return self._get_loc(key, axis=axis)

...site-packages/pandas/core/indexing.py in _validate_integer(self, key, axis)
   2061         len_axis = len(self.obj._get_axis(axis))
   2062         if key >= len_axis or key < -len_axis:
-> 2063             raise IndexError("single positional indexer is out-of-bounds")
   2064 
   2065     def _getitem_tuple(self, tup: Tuple):

IndexError: single positional indexer is out-of-bounds

I am not sure what's wrong with hits particular set of predictions (at least from metrics perspective ;-) )

Context: SacredObserver but probably related to Neptune itself.

Whenever I use logger level of 10, I get core dumped error. logger level of >=20 works fine. Stacktrace:

(...)
I1010 13:52:19.454688 140211946456832 connectionpool.py:393] https://ui.neptune.ml:443 "POST /api/backend/v1/channels/system?experimentId=26f075d2-2412-4971-81c9-c5322745a03c HTTP/1.1" 200 82
I1010 13:52:19.455582 140211946456832 client.py:239] createSystemChannel({'experimentId': '26f075d2-2412-4971-81c9-c5322745a03c', 'channelToCreate': ChannelParams(channelType='text', name='stderr')})
I1010 13:52:19.456417 140211946456832 client.py:239] createSystemChannel({'experimentId': '26f075d2-2412-4971-81c9-c5322745a03c', 'channelToCreate': ChannelParams(channelType='text', name='stderr')})
Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007f852299f700 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 300 in wait
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/queue.py", line 179 in get
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 111 in run
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 885 in _bootstrap

Current thread 0x00007f85a3430700 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/posixpath.py", line 42 in _get_sep
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/posixpath.py", line 147 in basename
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 327 in __init__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1488 in makeRecord
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1518 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/urllib3/connectionpool.py", line 393 in _make_request
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600 in urlopen
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests/adapters.py", line 449 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/requests/sessions.py", line 646 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/requests_client.py", line 392 in result
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 290 in _get_incoming_response
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 124 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/http_future.py", line 197 in response
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1037 in emit
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 797 in _log_to_stderr
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 825 in emit
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 905 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 465 in create_system_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 1110 in _create_channel
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/channels/channels_values_sender.py", line 66 in send
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/channel_writer.py", line 52 in write
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/streams/stdstream_uploader.py", line 33 in write
  ...

Thread 0x00007f85a2c2f700 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/ssl.py", line 913 in read
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/ssl.py", line 1037 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_socket.py", line 84 in _recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_socket.py", line 102 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 452 in _recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 396 in recv_strict
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 309 in recv_header
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_abnf.py", line 361 in recv_frame
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 377 in recv_frame
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 343 in recv_data_frame
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/websocket/_core.py", line 330 in recv_data
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/websockets/websocket_client_adapter.py", line 34 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/websockets/reconnecting_websocket.py", line 43 in recv
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/threads/aborting_thread.py", line 34 in run
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 917 in _bootstrap_inner
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/threading.py", line 885 in _bootstrap

Thread 0x00007f85a7844740 (most recent call first):
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 854 in acquire
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 903 in handle
  File "/home/wojciech/.local/lib/python3.7/site-packages/absl/logging/__init__.py", line 891 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1591 in callHandlers
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1529 in handle
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1519 in _log
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/logging/__init__.py", line 1371 in debug
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/bravado/client.py", line 239 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/backends/hosted_neptune_backend.py", line 585 in create_hardware_metric
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/utils.py", line 188 in wrapper
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/hardware/metrics/service/metric_service_factory.py", line 40 in create
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/execution/execution_context.py", line 151 in _run_hardware_metrics_reporting_thread
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/internal/execution/execution_context.py", line 87 in start
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/experiments.py", line 987 in _start
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/projects.py", line 431 in create_experiment
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/neptune/__init__.py", line 230 in create_experiment
  File "/home/wojciech/projects/magellan/sacred_utils.py", line 134 in started_event
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/run.py", line 326 in _emit_started
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/run.py", line 230 in __call__
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 212 in run
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 263 in run_commandline
  File "/home/wojciech/miniconda3/envs/magellan-mbbl/lib/python3.7/site-packages/sacred/experiment.py", line 140 in automain
  File "main.py", line 716 in <module>
[1]    10623 abort (core dumped)  main.py

I have noticed that NeptuneObserver doesn't respect the git hash and the source file names that sacred retrieved but has its own methods to retrieve them. In general, this is wrong because it breaks the RunObserver contract from the user point of view. Thus surprised me a lot and I spent o lot of time to understand that.

Although, I could live with that but in certain situations, NeptuneObserver fails to get the proper git commit hash while sacred doesn't. I couldn't track down the exact reason for that but it happens when

1. I run my experiment script (git repo) from a different script
2. my working directory (outside git repo) is different from my experiment script directory (git repo)

A similar situation was with sources. One solution was to set manually the base_dir in NeptuneObserver (but notice: sacred has its own base_dir mechanism - why not to use it?). But when I set my base_dir to e.g. /home/name/x/y, then the whole tree is then in the sources (starting with home), which is not ideal.

A second (not very elegant) workaround is to use os.chdir() - then the sources are gathered correctly but this doesn't affect git hash which is still missing, unfortunately.

The parameter values of None, False and True get converted to strings of 'None', 'False' and 'True' when downloaded neptune.get_experiment(). Easy to work-around but irritating.

I am not sure whether this is NeptuneObserver issue or Neptune issue since I am working solely with sacred.