Add files using upload-large-folder tool

stanza/demo/Stanza_CoreNLP_Interface.ipynb

@@ -0,0 +1,485 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "name": "Stanza-CoreNLP-Interface.ipynb",
+      "provenance": [],
+      "collapsed_sections": [],
+      "toc_visible": true
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2-4lzQTC9yxG",
+        "colab_type": "text"
+      },
+      "source": [
+        "# Stanza: A Tutorial on the Python CoreNLP Interface\n",
+        "\n",
+        "![Latest Version](https://img.shields.io/pypi/v/stanza.svg?colorB=bc4545)\n",
+        "![Python Versions](https://img.shields.io/pypi/pyversions/stanza.svg?colorB=bc4545)\n",
+        "\n",
+        "While the Stanza library implements accurate neural network modules for basic functionalities such as part-of-speech tagging and dependency parsing, the [Stanford CoreNLP Java library](https://stanfordnlp.github.io/CoreNLP/) has been developed for years and offers more complementary features such as coreference resolution and relation extraction. To unlock these features, the Stanza library also offers an officially maintained Python interface to the CoreNLP Java library. This interface allows you to get NLP anntotations from CoreNLP by writing native Python code.\n",
+        "\n",
+        "\n",
+        "This tutorial walks you through the installation, setup and basic usage of this Python CoreNLP interface. If you want to learn how to use the neural network components in Stanza, please refer to other tutorials."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "YpKwWeVkASGt",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 1. Installation\n",
+        "\n",
+        "Before the installation starts, please make sure that you have Python 3 and Java installed on your computer. Since Colab already has them installed, we'll skip this procedure in this notebook."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "k1Az2ECuAfG8",
+        "colab_type": "text"
+      },
+      "source": [
+        "### Installing Stanza\n",
+        "\n",
+        "Installing and importing Stanza are as simple as running the following commands:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "xiFwYAgW4Mss",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Install stanza; note that the prefix \"!\" is not needed if you are running in a terminal\n",
+        "!pip install stanza\n",
+        "\n",
+        "# Import stanza\n",
+        "import stanza"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "2zFvaA8_A32_",
+        "colab_type": "text"
+      },
+      "source": [
+        "### Setting up Stanford CoreNLP\n",
+        "\n",
+        "In order for the interface to work, the Stanford CoreNLP library has to be installed and a `CORENLP_HOME` environment variable has to be pointed to the installation location.\n",
+        "\n",
+        "Here we are going to show you how to download and install the CoreNLP library on your machine, with Stanza's installation command:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "MgK6-LPV-OdA",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Download the Stanford CoreNLP package with Stanza's installation command\n",
+        "# This'll take several minutes, depending on the network speed\n",
+        "corenlp_dir = './corenlp'\n",
+        "stanza.install_corenlp(dir=corenlp_dir)\n",
+        "\n",
+        "# Set the CORENLP_HOME environment variable to point to the installation location\n",
+        "import os\n",
+        "os.environ[\"CORENLP_HOME\"] = corenlp_dir"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Jdq8MT-NAhKj",
+        "colab_type": "text"
+      },
+      "source": [
+        "That's all for the installation! 🎉  We can now double check if the installation is successful by listing files in the CoreNLP directory. You should be able to see a number of `.jar` files by running the following command:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "K5eIOaJp_tuo",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Examine the CoreNLP installation folder to make sure the installation is successful\n",
+        "!ls $CORENLP_HOME"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "S0xb9BHt__gx",
+        "colab_type": "text"
+      },
+      "source": [
+        "**Note 1**:\n",
+        "If you are want to use the interface in a terminal (instead of a Colab notebook), you can properly set the `CORENLP_HOME` environment variable with:\n",
+        "\n",
+        "```bash\n",
+        "export CORENLP_HOME=path_to_corenlp_dir\n",
+        "```\n",
+        "\n",
+        "Here we instead set this variable with the Python `os` library, simply because `export` command is not well-supported in Colab notebook.\n",
+        "\n",
+        "\n",
+        "**Note 2**:\n",
+        "The `stanza.install_corenlp()` function is only available since Stanza v1.1.1. If you are using an earlier version of Stanza, please check out our [manual installation page](https://stanfordnlp.github.io/stanza/client_setup.html#manual-installation) for how to install CoreNLP on your computer.\n",
+        "\n",
+        "**Note 3**:\n",
+        "Besides the installation function, we also provide a `stanza.download_corenlp_models()` function to help you download additional CoreNLP models for different languages that are not shipped with the default installation. Check out our [automatic installation website page](https://stanfordnlp.github.io/stanza/client_setup.html#automated-installation) for more information on how to use it."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "xJsuO6D8D05q",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 2. Annotating Text with CoreNLP Interface"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "dZNHxXHkH1K2",
+        "colab_type": "text"
+      },
+      "source": [
+        "### Constructing CoreNLPClient\n",
+        "\n",
+        "At a high level, the CoreNLP Python interface works by first starting a background Java CoreNLP server process, and then initializing a client instance in Python which can pass the text to the background server process, and accept the returned annotation results.\n",
+        "\n",
+        "We wrap these functionalities in a `CoreNLPClient` class. Therefore, we need to start by importing this class from Stanza."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "LS4OKnqJ8wui",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Import client module\n",
+        "from stanza.server import CoreNLPClient"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "WP4Dz6PIJHeL",
+        "colab_type": "text"
+      },
+      "source": [
+        "After the import is done, we can construct a `CoreNLPClient` instance. The constructor method takes a Python list of annotator names as argument. Here let's explore some basic annotators including tokenization, sentence split, part-of-speech tagging, lemmatization and named entity recognition (NER). \n",
+        "\n",
+        "Additionally, the client constructor accepts a `memory` argument, which specifies how much memory will be allocated to the background Java process. An `endpoint` option can be used to specify a port number used by the communication between the server and the client. The default port is 9000. However, since this port is pre-occupied by a system process in Colab, we'll manually set it to 9001 in the following example.\n",
+        "\n",
+        "Also, here we manually set `be_quiet=True` to avoid an IO issue in colab notebook. You should be able to use `be_quiet=False` on your own computer, which will print detailed logging information from CoreNLP during usage.\n",
+        "\n",
+        "For more options in constructing the clients, please refer to the [CoreNLP Client Options List](https://stanfordnlp.github.io/stanza/corenlp_client.html#corenlp-client-options)."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "mbOBugvd9JaM",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Construct a CoreNLPClient with some basic annotators, a memory allocation of 4GB, and port number 9001\n",
+        "client = CoreNLPClient(\n",
+        "    annotators=['tokenize','ssplit', 'pos', 'lemma', 'ner'], \n",
+        "    memory='4G', \n",
+        "    endpoint='http://localhost:9001',\n",
+        "    be_quiet=True)\n",
+        "print(client)\n",
+        "\n",
+        "# Start the background server and wait for some time\n",
+        "# Note that in practice this is totally optional, as by default the server will be started when the first annotation is performed\n",
+        "client.start()\n",
+        "import time; time.sleep(10)"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "kgTiVjNydmIW",
+        "colab_type": "text"
+      },
+      "source": [
+        "After the above code block finishes executing, if you print the background processes, you should be able to find the Java CoreNLP server running."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "spZrJ-oFdkdF",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Print background processes and look for java\n",
+        "# You should be able to see a StanfordCoreNLPServer java process running in the background\n",
+        "!ps -o pid,cmd | grep java"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "KxJeJ0D2LoOs",
+        "colab_type": "text"
+      },
+      "source": [
+        "### Annotating Text\n",
+        "\n",
+        "Annotating a piece of text is as simple as passing the text into an `annotate` function of the client object. After the annotation is complete, a `Document`  object will be returned with all annotations.\n",
+        "\n",
+        "Note that although in general annotations are very fast, the first annotation might take a while to complete in the notebook. Please stay patient."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "s194RnNg5z95",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Annotate some text\n",
+        "text = \"Albert Einstein was a German-born theoretical physicist. He developed the theory of relativity.\"\n",
+        "document = client.annotate(text)\n",
+        "print(type(document))"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "semmA3e0TcM1",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 3. Accessing Annotations\n",
+        "\n",
+        "Annotations can be accessed from the returned `Document` object.\n",
+        "\n",
+        "A `Document` contains a list of `Sentence`s, which contain a list of `Token`s. Here let's first explore the annotations stored in all tokens."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "lIO4B5d6Rk4I",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Iterate over all tokens in all sentences, and print out the word, lemma, pos and ner tags\n",
+        "print(\"{:12s}\\t{:12s}\\t{:6s}\\t{}\".format(\"Word\", \"Lemma\", \"POS\", \"NER\"))\n",
+        "\n",
+        "for i, sent in enumerate(document.sentence):\n",
+        "    print(\"[Sentence {}]\".format(i+1))\n",
+        "    for t in sent.token:\n",
+        "        print(\"{:12s}\\t{:12s}\\t{:6s}\\t{}\".format(t.word, t.lemma, t.pos, t.ner))\n",
+        "    print(\"\")"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "msrJfvu8VV9m",
+        "colab_type": "text"
+      },
+      "source": [
+        "Alternatively, you can also browse the NER results by iterating over entity mentions over the sentences. For example:"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "ezEjc9LeV2Xs",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Iterate over all detected entity mentions\n",
+        "print(\"{:30s}\\t{}\".format(\"Mention\", \"Type\"))\n",
+        "\n",
+        "for sent in document.sentence:\n",
+        "    for m in sent.mentions:\n",
+        "        print(\"{:30s}\\t{}\".format(m.entityMentionText, m.entityType))"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "ueGzBZ3hWzkN",
+        "colab_type": "text"
+      },
+      "source": [
+        "To print all annotations a sentence, token or mention has, you can simply print the corresponding obejct."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "4_S8o2BHXIed",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Print annotations of a token\n",
+        "print(document.sentence[0].token[0])\n",
+        "\n",
+        "# Print annotations of a mention\n",
+        "print(document.sentence[0].mentions[0])"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "Qp66wjZ10xia",
+        "colab_type": "text"
+      },
+      "source": [
+        "**Note**: Since the Stanza CoreNLP client interface simply ports the CoreNLP annotation results to native Python objects, for a comprehensive lists of available annotators and how their annotation results can be accessed, you will need to visit the [Stanford CoreNLP website](https://stanfordnlp.github.io/CoreNLP/)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "IPqzMK90X0w3",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 4. Shutting Down the CoreNLP Server\n",
+        "\n",
+        "To shut down the background CoreNLP server process, simply call the `stop` function of the client. Note that once a server is shutdown, you'll have to restart the server with the `start()` function before any annotation is requested."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "xrJq8lZ3Nw7b",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "# Shut down the background CoreNLP server\n",
+        "client.stop()\n",
+        "\n",
+        "time.sleep(10)\n",
+        "!ps -o pid,cmd | grep java"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "23Vwa_ifYfF7",
+        "colab_type": "text"
+      },
+      "source": [
+        "### More Information\n",
+        "\n",
+        "For more information on how to use the `CoreNLPClient`, please go to the [CoreNLPClient documentation page](https://stanfordnlp.github.io/stanza/corenlp_client.html)."
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "YUrVT6kA_Bzx",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 5. Simplifying Client Usage with the Python `with` statement\n",
+        "\n",
+        "In the above demo, we explicitly called the `client.start()` and `client.stop()` functions to start and stop a client-server connection. However, doing this in practice is usually suboptimal, since you may forget to call the `stop()` function at the end, resulting in an unused server process occupying your machine memory.\n",
+        "\n",
+        "To solve is, a simple solution is to use the client interface with the [Python `with` statement](https://docs.python.org/3/reference/compound_stmts.html#the-with-statement). The `with` statement provides an elegant way to automatically start and stop the server process in your Python program, without you needing to worry about this. The following code snippet demonstrates how to establish a client, annotate an example text and then stop the server with a simple `with` statement. Note that we **always recommend** you to use the `with` statement when working with the Stanza CoreNLP client interface."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "metadata": {
+        "id": "H0ct2-R4AvJh",
+        "colab_type": "code",
+        "colab": {}
+      },
+      "source": [
+        "print(\"Starting a server with the Python \\\"with\\\" statement...\")\n",
+        "with CoreNLPClient(annotators=['tokenize','ssplit', 'pos', 'lemma', 'ner'], \n",
+        "                   memory='4G', endpoint='http://localhost:9001', be_quiet=True) as client:\n",
+        "    text = \"Albert Einstein was a German-born theoretical physicist.\"\n",
+        "    document = client.annotate(text)\n",
+        "\n",
+        "    print(\"{:30s}\\t{}\".format(\"Mention\", \"Type\"))\n",
+        "    for sent in document.sentence:\n",
+        "        for m in sent.mentions:\n",
+        "            print(\"{:30s}\\t{}\".format(m.entityMentionText, m.entityType))\n",
+        "\n",
+        "print(\"\\nThe server should be stopped upon exit from the \\\"with\\\" statement.\")"
+      ],
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {
+        "id": "W435Lwc4YqKb",
+        "colab_type": "text"
+      },
+      "source": [
+        "## 6. Other Resources\n",
+        "\n",
+        "- [Stanza Homepage](https://stanfordnlp.github.io/stanza/)\n",
+        "- [FAQs](https://stanfordnlp.github.io/stanza/faq.html)\n",
+        "- [GitHub Repo](https://github.com/stanfordnlp/stanza)\n",
+        "- [Reporting Issues](https://github.com/stanfordnlp/stanza/issues)\n"
+      ]
+    }
+  ]
+}

stanza/demo/en_test.conllu.txt

@@ -0,0 +1,79 @@
+# newdoc id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200
+# sent_id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200-0001
+# newpar id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200-p0001
+# text = What if Google Morphed Into GoogleOS?
+1	What	what	PRON	WP	PronType=Int	0	root	0:root	_
+2	if	if	SCONJ	IN	_	4	mark	4:mark	_
+3	Google	Google	PROPN	NNP	Number=Sing	4	nsubj	4:nsubj	_
+4	Morphed	morph	VERB	VBD	Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin	1	advcl	1:advcl:if	_
+5	Into	into	ADP	IN	_	6	case	6:case	_
+6	GoogleOS	GoogleOS	PROPN	NNP	Number=Sing	4	obl	4:obl:into	SpaceAfter=No
+7	?	?	PUNCT	.	_	4	punct	4:punct	_
+
+# sent_id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200-0002
+# text = What if Google expanded on its search-engine (and now e-mail) wares into a full-fledged operating system?
+1	What	what	PRON	WP	PronType=Int	0	root	0:root	_
+2	if	if	SCONJ	IN	_	4	mark	4:mark	_
+3	Google	Google	PROPN	NNP	Number=Sing	4	nsubj	4:nsubj	_
+4	expanded	expand	VERB	VBD	Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin	1	advcl	1:advcl:if	_
+5	on	on	ADP	IN	_	15	case	15:case	_
+6	its	its	PRON	PRP$	Gender=Neut|Number=Sing|Person=3|Poss=Yes|PronType=Prs	15	nmod:poss	15:nmod:poss	_
+7	search	search	NOUN	NN	Number=Sing	9	compound	9:compound	SpaceAfter=No
+8	-	-	PUNCT	HYPH	_	9	punct	9:punct	SpaceAfter=No
+9	engine	engine	NOUN	NN	Number=Sing	15	compound	15:compound	_
+10	(	(	PUNCT	-LRB-	_	9	punct	9:punct	SpaceAfter=No
+11	and	and	CCONJ	CC	_	13	cc	13:cc	_
+12	now	now	ADV	RB	_	13	advmod	13:advmod	_
+13	e-mail	e-mail	NOUN	NN	Number=Sing	9	conj	9:conj:and|15:compound	SpaceAfter=No
+14	)	)	PUNCT	-RRB-	_	15	punct	15:punct	_
+15	wares	wares	NOUN	NNS	Number=Plur	4	obl	4:obl:on	_
+16	into	into	ADP	IN	_	22	case	22:case	_
+17	a	a	DET	DT	Definite=Ind|PronType=Art	22	det	22:det	_
+18	full	full	ADV	RB	_	20	advmod	20:advmod	SpaceAfter=No
+19	-	-	PUNCT	HYPH	_	20	punct	20:punct	SpaceAfter=No
+20	fledged	fledged	ADJ	JJ	Degree=Pos	22	amod	22:amod	_
+21	operating	operating	NOUN	NN	Number=Sing	22	compound	22:compound	_
+22	system	system	NOUN	NN	Number=Sing	4	obl	4:obl:into	SpaceAfter=No
+23	?	?	PUNCT	.	_	4	punct	4:punct	_
+
+# sent_id = weblog-blogspot.com_zentelligence_20040423000200_ENG_20040423_000200-0003
+# text = [via Microsoft Watch from Mary Jo Foley ]
+1	[	[	PUNCT	-LRB-	_	4	punct	4:punct	SpaceAfter=No
+2	via	via	ADP	IN	_	4	case	4:case	_
+3	Microsoft	Microsoft	PROPN	NNP	Number=Sing	4	compound	4:compound	_
+4	Watch	Watch	PROPN	NNP	Number=Sing	0	root	0:root	_
+5	from	from	ADP	IN	_	6	case	6:case	_
+6	Mary	Mary	PROPN	NNP	Number=Sing	4	nmod	4:nmod:from	_
+7	Jo	Jo	PROPN	NNP	Number=Sing	6	flat	6:flat	_
+8	Foley	Foley	PROPN	NNP	Number=Sing	6	flat	6:flat	_
+9	]	]	PUNCT	-RRB-	_	4	punct	4:punct	_
+
+# newdoc id = weblog-blogspot.com_marketview_20050511222700_ENG_20050511_222700
+# sent_id = weblog-blogspot.com_marketview_20050511222700_ENG_20050511_222700-0001
+# newpar id = weblog-blogspot.com_marketview_20050511222700_ENG_20050511_222700-p0001
+# text = (And, by the way, is anybody else just a little nostalgic for the days when that was a good thing?)
+1	(	(	PUNCT	-LRB-	_	14	punct	14:punct	SpaceAfter=No
+2	And	and	CCONJ	CC	_	14	cc	14:cc	SpaceAfter=No
+3	,	,	PUNCT	,	_	14	punct	14:punct	_
+4	by	by	ADP	IN	_	6	case	6:case	_
+5	the	the	DET	DT	Definite=Def|PronType=Art	6	det	6:det	_
+6	way	way	NOUN	NN	Number=Sing	14	obl	14:obl:by	SpaceAfter=No
+7	,	,	PUNCT	,	_	14	punct	14:punct	_
+8	is	be	AUX	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	14	cop	14:cop	_
+9	anybody	anybody	PRON	NN	Number=Sing	14	nsubj	14:nsubj	_
+10	else	else	ADJ	JJ	Degree=Pos	9	amod	9:amod	_
+11	just	just	ADV	RB	_	13	advmod	13:advmod	_
+12	a	a	DET	DT	Definite=Ind|PronType=Art	13	det	13:det	_
+13	little	little	ADJ	JJ	Degree=Pos	14	obl:npmod	14:obl:npmod	_
+14	nostalgic	nostalgic	NOUN	NN	Number=Sing	0	root	0:root	_
+15	for	for	ADP	IN	_	17	case	17:case	_
+16	the	the	DET	DT	Definite=Def|PronType=Art	17	det	17:det	_
+17	days	day	NOUN	NNS	Number=Plur	14	nmod	14:nmod:for|23:obl:npmod	_
+18	when	when	ADV	WRB	PronType=Rel	23	advmod	17:ref	_
+19	that	that	PRON	DT	Number=Sing|PronType=Dem	23	nsubj	23:nsubj	_
+20	was	be	AUX	VBD	Mood=Ind|Number=Sing|Person=3|Tense=Past|VerbForm=Fin	23	cop	23:cop	_
+21	a	a	DET	DT	Definite=Ind|PronType=Art	23	det	23:det	_
+22	good	good	ADJ	JJ	Degree=Pos	23	amod	23:amod	_
+23	thing	thing	NOUN	NN	Number=Sing	17	acl:relcl	17:acl:relcl	SpaceAfter=No
+24	?	?	PUNCT	.	_	14	punct	14:punct	SpaceAfter=No
+25	)	)	PUNCT	-RRB-	_	14	punct	14:punct	_

stanza/demo/semgrex visualization.ipynb

@@ -0,0 +1,367 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2787d5f5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import stanza\n",
+    "from stanza.server.semgrex import Semgrex\n",
+    "from stanza.models.common.constant import is_right_to_left\n",
+    "import spacy\n",
+    "from spacy import displacy\n",
+    "from spacy.tokens import Doc\n",
+    "from IPython.display import display, HTML\n",
+    "\n",
+    "\n",
+    "\"\"\"\n",
+    "IMPORTANT: For the code in this module to run, you must have corenlp and Java installed on your machine. Additionally,\n",
+    "set an environment variable CLASSPATH equal to the path of your corenlp directory.\n",
+    "\n",
+    "Example: CLASSPATH=C:\\\\Users\\\\Alex\\\\PycharmProjects\\\\pythonProject\\\\stanford-corenlp-4.5.0\\\\stanford-corenlp-4.5.0\\\\*\n",
+    "\"\"\"\n",
+    "\n",
+    "%env CLASSPATH=C:\\\\stanford-corenlp-4.5.2\\\\stanford-corenlp-4.5.2\\\\*\n",
+    "def get_sentences_html(doc, language):\n",
+    "    \"\"\"\n",
+    "    Returns a list of the HTML strings of the dependency visualizations of a given stanza doc object.\n",
+    "\n",
+    "    The 'language' arg is the two-letter language code for the document to be processed.\n",
+    "\n",
+    "    First converts the stanza doc object to a spacy doc object and uses displacy to generate an HTML\n",
+    "    string for each sentence of the doc object.\n",
+    "    \"\"\"\n",
+    "    html_strings = []\n",
+    "\n",
+    "    # blank model - we don't use any of the model features, just the visualization\n",
+    "    nlp = spacy.blank(\"en\")\n",
+    "    sentences_to_visualize = []\n",
+    "    for sentence in doc.sentences:\n",
+    "        words, lemmas, heads, deps, tags = [], [], [], [], []\n",
+    "        if is_right_to_left(language):  # order of words displayed is reversed, dependency arcs remain intact\n",
+    "            sent_len = len(sentence.words)\n",
+    "            for word in reversed(sentence.words):\n",
+    "                words.append(word.text)\n",
+    "                lemmas.append(word.lemma)\n",
+    "                deps.append(word.deprel)\n",
+    "                tags.append(word.upos)\n",
+    "                if word.head == 0:  # spaCy head indexes are formatted differently than that of Stanza\n",
+    "                    heads.append(sent_len - word.id)\n",
+    "                else:\n",
+    "                    heads.append(sent_len - word.head)\n",
+    "        else:  # left to right rendering\n",
+    "            for word in sentence.words:\n",
+    "                words.append(word.text)\n",
+    "                lemmas.append(word.lemma)\n",
+    "                deps.append(word.deprel)\n",
+    "                tags.append(word.upos)\n",
+    "                if word.head == 0:\n",
+    "                    heads.append(word.id - 1)\n",
+    "                else:\n",
+    "                    heads.append(word.head - 1)\n",
+    "        document_result = Doc(nlp.vocab, words=words, lemmas=lemmas, heads=heads, deps=deps, pos=tags)\n",
+    "        sentences_to_visualize.append(document_result)\n",
+    "\n",
+    "    for line in sentences_to_visualize:  # render all sentences through displaCy\n",
+    "        html_strings.append(displacy.render(line, style=\"dep\",\n",
+    "                                            options={\"compact\": True, \"word_spacing\": 30, \"distance\": 100,\n",
+    "                                                     \"arrow_spacing\": 20}, jupyter=False))\n",
+    "    return html_strings\n",
+    "\n",
+    "\n",
+    "def find_nth(haystack, needle, n):\n",
+    "    \"\"\"\n",
+    "    Returns the starting index of the nth occurrence of the substring 'needle' in the string 'haystack'.\n",
+    "    \"\"\"\n",
+    "    start = haystack.find(needle)\n",
+    "    while start >= 0 and n > 1:\n",
+    "        start = haystack.find(needle, start + len(needle))\n",
+    "        n -= 1\n",
+    "    return start\n",
+    "\n",
+    "\n",
+    "def round_base(num, base=10):\n",
+    "    \"\"\"\n",
+    "    Rounding a number to its nearest multiple of the base. round_base(49.2, base=50) = 50.\n",
+    "    \"\"\"\n",
+    "    return base * round(num/base)\n",
+    "\n",
+    "\n",
+    "def process_sentence_html(orig_html, semgrex_sentence):\n",
+    "    \"\"\"\n",
+    "    Takes a semgrex sentence object and modifies the HTML of the original sentence's deprel visualization,\n",
+    "    highlighting words involved in the search queries and adding the label of the word inside of the semgrex match.\n",
+    "\n",
+    "    Returns the modified html string of the sentence's deprel visualization.\n",
+    "    \"\"\"\n",
+    "    tracker = {}  # keep track of which words have multiple labels\n",
+    "    DEFAULT_TSPAN_COUNT = 2  # the original displacy html assigns two <tspan> objects per <text> object\n",
+    "    CLOSING_TSPAN_LEN = 8  # </tspan> is 8 chars long\n",
+    "    colors = ['#4477AA', '#66CCEE', '#228833', '#CCBB44', '#EE6677', '#AA3377', '#BBBBBB']\n",
+    "    css_bolded_class = \"<style> .bolded{font-weight: bold;} </style>\\n\"\n",
+    "    found_index = orig_html.find(\"\\n\")  # returns index where the opening <svg> ends\n",
+    "    # insert the new style class into html string\n",
+    "    orig_html = orig_html[: found_index + 1] + css_bolded_class + orig_html[found_index + 1:]\n",
+    "\n",
+    "    # Add color to words in the match, bold words in the match\n",
+    "    for query in semgrex_sentence.result:\n",
+    "        for i, match in enumerate(query.match):\n",
+    "            color = colors[i]\n",
+    "            paired_dy = 2\n",
+    "            for node in match.node:\n",
+    "                name, match_index = node.name, node.matchIndex\n",
+    "                # edit existing <tspan> to change color and bold the text\n",
+    "                start = find_nth(orig_html, \"<text\", match_index)  # finds start of svg <text> of interest\n",
+    "                if match_index not in tracker:  # if we've already bolded and colored, keep the first color\n",
+    "                    tspan_start = orig_html.find(\"<tspan\",\n",
+    "                                                 start)  # finds start of the first svg <tspan> inside of the <text>\n",
+    "                    tspan_end = orig_html.find(\"</tspan>\", start)  # finds start of the end of the above <tspan>\n",
+    "                    tspan_substr = orig_html[tspan_start: tspan_end + CLOSING_TSPAN_LEN + 1] + \"\\n\"\n",
+    "                    # color words in the hit and bold words in the hit\n",
+    "                    edited_tspan = tspan_substr.replace('class=\"displacy-word\"', 'class=\"bolded\"').replace(\n",
+    "                        'fill=\"currentColor\"', f'fill=\"{color}\"')\n",
+    "                    # insert edited <tspan> object into html string\n",
+    "                    orig_html = orig_html[: tspan_start] + edited_tspan + orig_html[tspan_end + CLOSING_TSPAN_LEN + 2:]\n",
+    "                    tracker[match_index] = DEFAULT_TSPAN_COUNT\n",
+    "\n",
+    "                # next, we have to insert the new <tspan> object for the label\n",
+    "                # Copy old <tspan> to copy formatting when creating new <tspan> later\n",
+    "                prev_tspan_start = find_nth(orig_html[start:], \"<tspan\",\n",
+    "                                            tracker[match_index] - 1) + start  # find the previous <tspan> start index\n",
+    "                prev_tspan_end = find_nth(orig_html[start:], \"</tspan>\",\n",
+    "                                          tracker[match_index] - 1) + start  # find the prev </tspan> start index\n",
+    "                prev_tspan = orig_html[prev_tspan_start: prev_tspan_end + CLOSING_TSPAN_LEN + 1]\n",
+    "\n",
+    "                # Find spot to insert new tspan\n",
+    "                closing_tspan_start = find_nth(orig_html[start:], \"</tspan>\", tracker[match_index]) + start\n",
+    "                up_to_new_tspan = orig_html[: closing_tspan_start + CLOSING_TSPAN_LEN + 1]\n",
+    "                rest_need_add_newline = orig_html[closing_tspan_start + CLOSING_TSPAN_LEN + 1:]\n",
+    "\n",
+    "                # Calculate proper x value in svg\n",
+    "                x_value_start = prev_tspan.find('x=\"')\n",
+    "                x_value_end = prev_tspan[x_value_start + 3:].find('\"') + 3  # 3 is the length of the 'x=\"' substring\n",
+    "                x_value = prev_tspan[x_value_start + 3: x_value_end + x_value_start]\n",
+    "\n",
+    "                # Calculate proper y value in svg\n",
+    "                DEFAULT_DY_VAL, dy = 2, 2\n",
+    "                if paired_dy != DEFAULT_DY_VAL and node == match.node[\n",
+    "                    1]:  # we're on the second node and need to adjust height to match the paired node\n",
+    "                    dy = paired_dy\n",
+    "                if node == match.node[0]:\n",
+    "                    paired_node_level = 2\n",
+    "                    if match.node[1].matchIndex in tracker:  # check if we need to adjust heights of labels\n",
+    "                        paired_node_level = tracker[match.node[1].matchIndex]\n",
+    "                        dif = tracker[match_index] - paired_node_level\n",
+    "                        if dif > 0:  # current node has more labels\n",
+    "                            paired_dy = DEFAULT_DY_VAL * dif + 1\n",
+    "                            dy = DEFAULT_DY_VAL\n",
+    "                        else:  # paired node has more labels, adjust this label down\n",
+    "                            dy = DEFAULT_DY_VAL * (abs(dif) + 1)\n",
+    "                            paired_dy = DEFAULT_DY_VAL\n",
+    "\n",
+    "                # Insert new <tspan> object\n",
+    "                new_tspan = f'  <tspan class=\"displacy-word\" dy=\"{dy}em\" fill=\"{color}\" x={x_value}>{name[: 3].title()}.</tspan>\\n'  # abbreviate label names to 3 chars\n",
+    "                orig_html = up_to_new_tspan + new_tspan + rest_need_add_newline\n",
+    "                tracker[match_index] += 1\n",
+    "    return orig_html\n",
+    "\n",
+    "\n",
+    "def render_html_strings(edited_html_strings):\n",
+    "    \"\"\"\n",
+    "    Renders the HTML to make the edits visible\n",
+    "    \"\"\"\n",
+    "    for html_string in edited_html_strings:\n",
+    "        display(HTML(html_string))\n",
+    "\n",
+    "\n",
+    "def visualize_search_doc(doc, semgrex_queries, lang_code, start_match=0, end_match=10):\n",
+    "    \"\"\"\n",
+    "    Visualizes the semgrex results of running semgrex search on a stanza doc object with the given list of\n",
+    "    semgrex queries. Returns a list of the edited HTML strings from the doc. Each element in the list represents\n",
+    "    the HTML to render one of the sentences in the document.\n",
+    "\n",
+    "    'lang_code' is the two-letter language abbreviation for the language that the stanza doc object is written in.\n",
+    "\n",
+    "\n",
+    "    'start_match' and 'end_match' determine which matches to visualize. Works similar to splices, so that\n",
+    "    start_match=0 and end_match=10 will display the first 10 semgrex matches.\n",
+    "    \"\"\"\n",
+    "    matches_count = 0  # Limits number of visualizations\n",
+    "    with Semgrex(classpath=\"$CLASSPATH\") as sem:\n",
+    "        edited_html_strings = []\n",
+    "        semgrex_results = sem.process(doc, *semgrex_queries)\n",
+    "        # one html string for each sentence\n",
+    "        unedited_html_strings = get_sentences_html(doc, lang_code)\n",
+    "        for i in range(len(unedited_html_strings)):\n",
+    "\n",
+    "            if matches_count >= end_match:  # we've collected enough matches, stop early\n",
+    "                break\n",
+    "\n",
+    "            # check if sentence has matches, if not then do not visualize\n",
+    "            has_none = True\n",
+    "            for query in semgrex_results.result[i].result:\n",
+    "                for match in query.match:\n",
+    "                    if match:\n",
+    "                        has_none = False\n",
+    "\n",
+    "            # Process HTML if queries have matches\n",
+    "            if not has_none:\n",
+    "                if start_match <= matches_count < end_match:\n",
+    "                    edited_string = process_sentence_html(unedited_html_strings[i], semgrex_results.result[i])\n",
+    "                    edited_string = adjust_dep_arrows(edited_string)\n",
+    "                    edited_html_strings.append(edited_string)\n",
+    "                matches_count += 1\n",
+    "\n",
+    "        render_html_strings(edited_html_strings)\n",
+    "    return edited_html_strings\n",
+    "\n",
+    "\n",
+    "def visualize_search_str(text, semgrex_queries, lang_code):\n",
+    "    \"\"\"\n",
+    "    Visualizes the deprel of the semgrex results from running semgrex search on a string with the given list of\n",
+    "    semgrex queries. Returns a list of the edited HTML strings. Each element in the list represents\n",
+    "    the HTML to render one of the sentences in the document.\n",
+    "\n",
+    "    Internally, this function converts the string into a stanza doc object before processing the doc object.\n",
+    "\n",
+    "    'lang_code' is the two-letter language abbreviation for the language that the stanza doc object is written in.\n",
+    "    \"\"\"\n",
+    "    nlp = stanza.Pipeline(lang_code, processors=\"tokenize, pos, lemma, depparse\")\n",
+    "    doc = nlp(text)\n",
+    "    return visualize_search_doc(doc, semgrex_queries, lang_code)\n",
+    "\n",
+    "\n",
+    "def adjust_dep_arrows(raw_html):\n",
+    "    \"\"\"\n",
+    "    The default spaCy dependency visualization has misaligned arrows.\n",
+    "    We fix arrows by aligning arrow ends and bodies to the word that they are directed to. If a word has an\n",
+    "    arrowhead that is pointing not directly on the word's center, align the arrowhead to match the center of the word.\n",
+    "\n",
+    "    returns the edited html with fixed arrow placement\n",
+    "    \"\"\"\n",
+    "    HTML_ARROW_BEGINNING = '<g class=\"displacy-arrow\">'\n",
+    "    HTML_ARROW_ENDING = \"</g>\"\n",
+    "    HTML_ARROW_ENDING_LEN = 6   # there are 2 newline chars after the arrow ending\n",
+    "    arrows_start_idx = find_nth(haystack=raw_html, needle='<g class=\"displacy-arrow\">', n=1)\n",
+    "    words_html, arrows_html = raw_html[: arrows_start_idx], raw_html[arrows_start_idx:]  # separate html for words and arrows\n",
+    "    final_html = words_html  # continually concatenate to this after processing each arrow\n",
+    "    arrow_number = 1  # which arrow we're editing (1-indexed)\n",
+    "    start_idx, end_of_class_idx = find_nth(haystack=arrows_html, needle=HTML_ARROW_BEGINNING, n=arrow_number), find_nth(arrows_html, HTML_ARROW_ENDING, arrow_number)\n",
+    "    while start_idx != -1:  # edit every arrow\n",
+    "        arrow_section = arrows_html[start_idx: end_of_class_idx + HTML_ARROW_ENDING_LEN]  # slice a single svg arrow object\n",
+    "        if arrow_section[-1] == \"<\":   # this is the last arrow in the HTML, don't cut the splice early\n",
+    "            arrow_section = arrows_html[start_idx:]\n",
+    "        edited_arrow_section = edit_dep_arrow(arrow_section)\n",
+    "\n",
+    "        final_html = final_html + edited_arrow_section  # continually update html with new arrow html until done\n",
+    "\n",
+    "        # Prepare for next iteration\n",
+    "        arrow_number += 1\n",
+    "        start_idx = find_nth(arrows_html, '<g class=\"displacy-arrow\">', n=arrow_number)\n",
+    "        end_of_class_idx = find_nth(arrows_html, \"</g>\", arrow_number)\n",
+    "    return final_html\n",
+    "\n",
+    "\n",
+    "def edit_dep_arrow(arrow_html):\n",
+    "    \"\"\"\n",
+    "    The formatting of a displacy arrow in svg is the following:\n",
+    "    <g class=\"displacy-arrow\">\n",
+    "        <path class=\"displacy-arc\" id=\"arrow-c628889ffbf343e3848193a08606f10a-0-0\" stroke-width=\"2px\" d=\"M70,352.0 C70,177.0 390.0,177.0 390.0,352.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
+    "        <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
+    "            <textPath xlink:href=\"#arrow-c628889ffbf343e3848193a08606f10a-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">csubj</textPath>\n",
+    "        </text>\n",
+    "        <path class=\"displacy-arrowhead\" d=\"M70,354.0 L62,342.0 78,342.0\" fill=\"currentColor\"/>\n",
+    "    </g>\n",
+    "\n",
+    "    We edit the 'd = ...' parts of the <path class ...> section to fix the arrow direction and length\n",
+    "\n",
+    "    returns the arrow_html with distances fixed\n",
+    "    \"\"\"\n",
+    "    WORD_SPACING = 50   # words start at x=50 and are separated by 100s so their x values are multiples of 50\n",
+    "    M_OFFSET = 4  # length of 'd=\"M' that we search for to extract the number from d=\"M70, for instance\n",
+    "    ARROW_PIXEL_SIZE = 4\n",
+    "    first_d_idx, second_d_idx = find_nth(arrow_html, 'd=\"M', 1), find_nth(arrow_html, 'd=\"M', 2)  # find where d=\"M starts\n",
+    "    first_d_cutoff, second_d_cutoff = arrow_html.find(\",\", first_d_idx), arrow_html.find(\",\", second_d_idx)  # isolate the number after 'M' e.g. 'M70'\n",
+    "    # gives svg x values of arrow body starting position and arrowhead position\n",
+    "    arrow_position, arrowhead_position = float(arrow_html[first_d_idx + M_OFFSET: first_d_cutoff]), float(arrow_html[second_d_idx + M_OFFSET: second_d_cutoff])\n",
+    "    # gives starting index of where 'fill=\"none\"' or 'fill=\"currentColor\"' begin, reference points to end the d= section\n",
+    "    first_fill_start_idx, second_fill_start_idx = find_nth(arrow_html, \"fill\", n=1), find_nth(arrow_html, \"fill\", n=3)\n",
+    "\n",
+    "    # isolate the d= ... section to edit\n",
+    "    first_d, second_d = arrow_html[first_d_idx: first_fill_start_idx], arrow_html[second_d_idx: second_fill_start_idx]\n",
+    "    first_d_split, second_d_split = first_d.split(\",\"), second_d.split(\",\")\n",
+    "\n",
+    "    if arrow_position == arrowhead_position:  # This arrow is incoming onto the word, center the arrow/head to word center\n",
+    "        corrected_arrow_pos = corrected_arrowhead_pos = round_base(arrow_position, base=WORD_SPACING)\n",
+    "\n",
+    "        # edit first_d  -- arrow body\n",
+    "        second_term = first_d_split[1].split(\" \")[0] + \" \" + str(corrected_arrow_pos)\n",
+    "        first_d = 'd=\"M' + str(corrected_arrow_pos) + \",\" + second_term + \",\" + \",\".join(first_d_split[2:])\n",
+    "\n",
+    "        # edit second_d  -- arrowhead\n",
+    "        second_term = second_d_split[1].split(\" \")[0] + \" L\" + str(corrected_arrowhead_pos - ARROW_PIXEL_SIZE)\n",
+    "        third_term = second_d_split[2].split(\" \")[0] + \" \" + str(corrected_arrowhead_pos + ARROW_PIXEL_SIZE)\n",
+    "        second_d = 'd=\"M' + str(corrected_arrowhead_pos) + \",\" + second_term + \",\" + third_term + \",\" + \",\".join(second_d_split[3:])\n",
+    "    else:  # This arrow is outgoing to another word, center the arrow/head to that word's center\n",
+    "        corrected_arrowhead_pos = round_base(arrowhead_position, base=WORD_SPACING)\n",
+    "\n",
+    "        # edit first_d -- arrow body\n",
+    "        third_term = first_d_split[2].split(\" \")[0] + \" \" + str(corrected_arrowhead_pos)\n",
+    "        fourth_term = first_d_split[3].split(\" \")[0] + \" \" + str(corrected_arrowhead_pos)\n",
+    "        terms = [first_d_split[0], first_d_split[1], third_term, fourth_term] + first_d_split[4:]\n",
+    "        first_d = \",\".join(terms)\n",
+    "\n",
+    "        # edit second_d -- arrow head\n",
+    "        first_term = f'd=\"M{corrected_arrowhead_pos}'\n",
+    "        second_term = second_d_split[1].split(\" \")[0] + \" L\" + str(corrected_arrowhead_pos - ARROW_PIXEL_SIZE)\n",
+    "        third_term = second_d_split[2].split(\" \")[0] + \" \" + str(corrected_arrowhead_pos + ARROW_PIXEL_SIZE)\n",
+    "        terms = [first_term, second_term, third_term] + second_d_split[3:]\n",
+    "        second_d = \",\".join(terms)\n",
+    "    # rebuild and return html\n",
+    "    return arrow_html[:first_d_idx] + first_d + \" \" + arrow_html[first_fill_start_idx:second_d_idx] + second_d + \" \" + arrow_html[second_fill_start_idx:]\n",
+    "\n",
+    "\n",
+    "def main():\n",
+    "    nlp = stanza.Pipeline(\"en\", processors=\"tokenize,pos,lemma,depparse\")\n",
+    "\n",
+    "    # doc = nlp(\"This a dummy sentence. Banning opal removed all artifact decks from the meta.  I miss playing lantern. This is a dummy sentence.\")\n",
+    "    doc = nlp(\"Banning opal removed artifact decks from the meta. Banning tennis resulted in players banning people.\")\n",
+    "    # A single result .result[i].result[j] is a list of matches for sentence i on semgrex query j.\n",
+    "    queries = [\"{pos:NN}=object <obl {}=action\",\n",
+    "               \"{cpos:NOUN}=thing <obj {cpos:VERB}=action\"]\n",
+    "    res = visualize_search_doc(doc, queries, \"en\")\n",
+    "    print(res[0])  # see the first sentence's deprel visualization HTML\n",
+    "    print(\"---------------------------------------\")\n",
+    "    print(res[1])  # second sentence's deprel visualization HTML\n",
+    "    return\n",
+    "\n",
+    "\n",
+    "if __name__ == '__main__':\n",
+    "    main()\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

stanza/stanza/__init__.py

@@ -0,0 +1,27 @@
+from stanza.pipeline.core import DownloadMethod, Pipeline
+from stanza.pipeline.multilingual import MultilingualPipeline
+from stanza.models.common.doc import Document
+from stanza.resources.common import download
+from stanza.resources.installation import install_corenlp, download_corenlp_models
+from stanza._version import __version__, __resources_version__
+
+import logging
+logger = logging.getLogger('stanza')
+
+# if the client application hasn't set the log level, we set it
+# ourselves to INFO
+if logger.level == 0:
+    logger.setLevel(logging.INFO)
+
+log_handler = logging.StreamHandler()
+log_formatter = logging.Formatter(fmt="%(asctime)s %(levelname)s: %(message)s",
+                              datefmt='%Y-%m-%d %H:%M:%S')
+log_handler.setFormatter(log_formatter)
+
+# also, if the client hasn't added any handlers for this logger
+# (or a default handler), we add a handler of our own
+#
+# client can later do
+#   logger.removeHandler(stanza.log_handler)
+if not logger.hasHandlers():
+    logger.addHandler(log_handler)

stanza/stanza/models/charlm.py

@@ -0,0 +1,357 @@
+"""
+Entry point for training and evaluating a character-level neural language model.
+"""
+
+import argparse
+from copy import copy
+import logging
+import lzma
+import math
+import os
+import random
+import time
+from types import GeneratorType
+import numpy as np
+import torch
+
+from stanza.models.common.char_model import build_charlm_vocab, CharacterLanguageModel, CharacterLanguageModelTrainer
+from stanza.models.common.vocab import CharVocab
+from stanza.models.common import utils
+from stanza.models import _training_logging
+
+logger = logging.getLogger('stanza')
+
+def repackage_hidden(h):
+    """Wraps hidden states in new Tensors,
+    to detach them from their history."""
+    if isinstance(h, torch.Tensor):
+        return h.detach()
+    else:
+        return tuple(repackage_hidden(v) for v in h)
+
+def batchify(data, bsz, device):
+    # Work out how cleanly we can divide the dataset into bsz parts.
+    nbatch = data.size(0) // bsz
+    # Trim off any extra elements that wouldn't cleanly fit (remainders).
+    data = data.narrow(0, 0, nbatch * bsz)
+    # Evenly divide the data across the bsz batches.
+    data = data.view(bsz, -1) # batch_first is True
+    data = data.to(device)
+    return data
+
+def get_batch(source, i, seq_len):
+    seq_len = min(seq_len, source.size(1) - 1 - i)
+    data = source[:, i:i+seq_len]
+    target = source[:, i+1:i+1+seq_len].reshape(-1)
+    return data, target
+
+def load_file(filename, vocab, direction):
+    with utils.open_read_text(filename) as fin:
+        data = fin.read()
+
+    idx = vocab['char'].map(data)
+    if direction == 'backward': idx = idx[::-1]
+    return torch.tensor(idx)
+
+def load_data(path, vocab, direction):
+    if os.path.isdir(path):
+        filenames = sorted(os.listdir(path))
+        for filename in filenames:
+            logger.info('Loading data from {}'.format(filename))
+            data = load_file(os.path.join(path, filename), vocab, direction)
+            yield data
+    else:
+        data = load_file(path, vocab, direction)
+        yield data
+
+def build_argparse():
+    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+    parser.add_argument('--train_file', type=str, help="Input plaintext file")
+    parser.add_argument('--train_dir', type=str, help="If non-empty, load from directory with multiple training files")
+    parser.add_argument('--eval_file', type=str, help="Input plaintext file for the dev/test set")
+    parser.add_argument('--shorthand', type=str, help="UD treebank shorthand")
+
+    parser.add_argument('--mode', default='train', choices=['train', 'predict'])
+    parser.add_argument('--direction', default='forward', choices=['forward', 'backward'], help="Forward or backward language model")
+    parser.add_argument('--forward', action='store_const', dest='direction', const='forward', help="Train a forward language model")
+    parser.add_argument('--backward', action='store_const', dest='direction', const='backward', help="Train a backward language model")
+
+    parser.add_argument('--char_emb_dim', type=int, default=100, help="Dimension of unit embeddings")
+    parser.add_argument('--char_hidden_dim', type=int, default=1024, help="Dimension of hidden units")
+    parser.add_argument('--char_num_layers', type=int, default=1, help="Layers of RNN in the language model")
+    parser.add_argument('--char_dropout', type=float, default=0.05, help="Dropout probability")
+    parser.add_argument('--char_unit_dropout', type=float, default=1e-5, help="Randomly set an input char to UNK during training")
+    parser.add_argument('--char_rec_dropout', type=float, default=0.0, help="Recurrent dropout probability")
+
+    parser.add_argument('--batch_size', type=int, default=100, help="Batch size to use")
+    parser.add_argument('--bptt_size', type=int, default=250, help="Sequence length to consider at a time")
+    parser.add_argument('--epochs', type=int, default=50, help="Total epochs to train the model for")
+    parser.add_argument('--max_grad_norm', type=float, default=0.25, help="Maximum gradient norm to clip to")
+    parser.add_argument('--lr0', type=float, default=5, help="Initial learning rate")
+    parser.add_argument('--anneal', type=float, default=0.25, help="Anneal the learning rate by this amount when dev performance deteriorate")
+    parser.add_argument('--patience', type=int, default=1, help="Patience for annealing the learning rate")
+    parser.add_argument('--weight_decay', type=float, default=0.0, help="Weight decay")
+    parser.add_argument('--momentum', type=float, default=0.0, help='Momentum for SGD.')
+    parser.add_argument('--cutoff', type=int, default=1000, help="Frequency cutoff for char vocab. By default we assume a very large corpus.")
+    
+    parser.add_argument('--report_steps', type=int, default=50, help="Update step interval to report loss")
+    parser.add_argument('--eval_steps', type=int, default=100000, help="Update step interval to run eval on dev; set to -1 to eval after each epoch")
+    parser.add_argument('--save_name', type=str, default=None, help="File name to save the model")
+    parser.add_argument('--vocab_save_name', type=str, default=None, help="File name to save the vocab")
+    parser.add_argument('--checkpoint_save_name', type=str, default=None, help="File name to save the most recent checkpoint")
+    parser.add_argument('--no_checkpoint', dest='checkpoint', action='store_false', help="Don't save checkpoints")
+    parser.add_argument('--save_dir', type=str, default='saved_models/charlm', help="Directory to save models in")
+    parser.add_argument('--summary', action='store_true', help='Use summary writer to record progress.')
+    utils.add_device_args(parser)
+    parser.add_argument('--seed', type=int, default=1234)
+
+    parser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name')
+    parser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+    return parser
+
+def build_model_filename(args):
+    if args['save_name']:
+        save_name = args['save_name']
+    else:
+        save_name = '{}_{}_charlm.pt'.format(args['shorthand'], args['direction'])
+    model_file = os.path.join(args['save_dir'], save_name)
+    return model_file
+
+def parse_args(args=None):
+    parser = build_argparse()
+
+    args = parser.parse_args(args=args)
+
+    if args.wandb_name:
+        args.wandb = True
+
+    args = vars(args)
+    return args
+
+def main(args=None):
+    args = parse_args(args=args)
+
+    utils.set_random_seed(args['seed'])
+
+    logger.info("Running {} character-level language model in {} mode".format(args['direction'], args['mode']))
+    
+    utils.ensure_dir(args['save_dir'])
+
+    if args['mode'] == 'train':
+        train(args)
+    else:
+        evaluate(args)
+
+def evaluate_epoch(args, vocab, data, model, criterion):
+    """
+    Run an evaluation over entire dataset.
+    """
+    model.eval()
+    device = next(model.parameters()).device
+    hidden = None
+    total_loss = 0
+    if isinstance(data, GeneratorType):
+        data = list(data)
+        assert len(data) == 1, 'Only support single dev/test file'
+        data = data[0]
+    batches = batchify(data, args['batch_size'], device)
+    with torch.no_grad():
+        for i in range(0, batches.size(1) - 1, args['bptt_size']):
+            data, target = get_batch(batches, i, args['bptt_size'])
+            lens = [data.size(1) for i in range(data.size(0))]
+
+            output, hidden, decoded = model.forward(data, lens, hidden)
+            loss = criterion(decoded.view(-1, len(vocab['char'])), target)
+            
+            hidden = repackage_hidden(hidden)
+            total_loss += data.size(1) * loss.data.item()
+    return total_loss / batches.size(1)
+
+def evaluate_and_save(args, vocab, data, trainer, best_loss, model_file, checkpoint_file, writer=None):
+    """
+    Run an evaluation over entire dataset, print progress and save the model if necessary.
+    """
+    start_time = time.time()
+    loss = evaluate_epoch(args, vocab, data, trainer.model, trainer.criterion)
+    ppl = math.exp(loss)
+    elapsed = int(time.time() - start_time)
+    # TODO: step the scheduler less often when the eval frequency is higher
+    previous_lr = get_current_lr(trainer, args)
+    trainer.scheduler.step(loss)
+    current_lr = get_current_lr(trainer, args)
+    if previous_lr != current_lr:
+        logger.info("Updating learning rate to %f", current_lr)
+    logger.info(
+        "| eval checkpoint @ global step {:10d} | time elapsed {:6d}s | loss {:5.2f} | ppl {:8.2f}".format(
+            trainer.global_step,
+            elapsed,
+            loss,
+            ppl,
+        )
+    )
+    if best_loss is None or loss < best_loss:
+        best_loss = loss
+        trainer.save(model_file, full=False)
+        logger.info('new best model saved at step {:10d}'.format(trainer.global_step))
+    if writer:
+        writer.add_scalar('dev_loss', loss, global_step=trainer.global_step)
+        writer.add_scalar('dev_ppl', ppl, global_step=trainer.global_step)
+    if checkpoint_file:
+        trainer.save(checkpoint_file, full=True)
+        logger.info('new checkpoint saved at step {:10d}'.format(trainer.global_step))
+
+    return loss, ppl, best_loss
+
+def get_current_lr(trainer, args):
+    return trainer.scheduler.state_dict().get('_last_lr', [args['lr0']])[0]
+
+def load_char_vocab(vocab_file):
+    return {'char': CharVocab.load_state_dict(torch.load(vocab_file, lambda storage, loc: storage, weights_only=True))}
+
+def train(args):
+    utils.log_training_args(args, logger)
+    model_file = build_model_filename(args)
+
+    vocab_file = args['save_dir'] + '/' + args['vocab_save_name'] if args['vocab_save_name'] is not None \
+        else '{}/{}_vocab.pt'.format(args['save_dir'], args['shorthand'])
+
+    if args['checkpoint']:
+        checkpoint_file = utils.checkpoint_name(args['save_dir'], model_file, args['checkpoint_save_name'])
+    else:
+        checkpoint_file = None
+
+    if os.path.exists(vocab_file):
+        logger.info('Loading existing vocab file')
+        vocab = load_char_vocab(vocab_file)
+    else:
+        logger.info('Building and saving vocab')
+        vocab = {'char': build_charlm_vocab(args['train_file'] if args['train_dir'] is None else args['train_dir'], cutoff=args['cutoff'])}
+        torch.save(vocab['char'].state_dict(), vocab_file)
+    logger.info("Training model with vocab size: {}".format(len(vocab['char'])))
+
+    if checkpoint_file and os.path.exists(checkpoint_file):
+        logger.info('Loading existing checkpoint: %s' % checkpoint_file)
+        trainer = CharacterLanguageModelTrainer.load(args, checkpoint_file, finetune=True)
+    else:
+        trainer = CharacterLanguageModelTrainer.from_new_model(args, vocab)
+
+    writer = None
+    if args['summary']:
+        from torch.utils.tensorboard import SummaryWriter
+        summary_dir = '{}/{}_summary'.format(args['save_dir'], args['save_name']) if args['save_name'] is not None \
+            else '{}/{}_{}_charlm_summary'.format(args['save_dir'], args['shorthand'], args['direction'])
+        writer = SummaryWriter(log_dir=summary_dir)
+    
+    # evaluate model within epoch if eval_interval is set
+    eval_within_epoch = False
+    if args['eval_steps'] > 0:
+        eval_within_epoch = True
+
+    if args['wandb']:
+        import wandb
+        wandb_name = args['wandb_name'] if args['wandb_name'] else '%s_%s_charlm' % (args['shorthand'], args['direction'])
+        wandb.init(name=wandb_name, config=args)
+        wandb.run.define_metric('best_loss', summary='min')
+        wandb.run.define_metric('ppl', summary='min')
+
+    device = next(trainer.model.parameters()).device
+
+    best_loss = None
+    start_epoch = trainer.epoch  # will default to 1 for a new trainer
+    for trainer.epoch in range(start_epoch, args['epochs']+1):
+        # load train data from train_dir if not empty, otherwise load from file
+        if args['train_dir'] is not None:
+            train_path = args['train_dir']
+        else:
+            train_path = args['train_file']
+        train_data = load_data(train_path, vocab, args['direction'])
+        dev_data = load_file(args['eval_file'], vocab, args['direction']) # dev must be a single file
+
+        # run over entire training set
+        for data_chunk in train_data:
+            batches = batchify(data_chunk, args['batch_size'], device)
+            hidden = None
+            total_loss = 0.0
+            total_batches = math.ceil((batches.size(1) - 1) / args['bptt_size'])
+            iteration, i = 0, 0
+            # over the data chunk
+            while i < batches.size(1) - 1 - 1:
+                trainer.model.train()
+                trainer.global_step += 1
+                start_time = time.time()
+                bptt = args['bptt_size'] if np.random.random() < 0.95 else args['bptt_size']/ 2.
+                # prevent excessively small or negative sequence lengths
+                seq_len = max(5, int(np.random.normal(bptt, 5)))
+                # prevent very large sequence length, must be <= 1.2 x bptt
+                seq_len = min(seq_len, int(args['bptt_size'] * 1.2))
+                data, target = get_batch(batches, i, seq_len)
+                lens = [data.size(1) for i in range(data.size(0))]
+                
+                trainer.optimizer.zero_grad()
+                output, hidden, decoded = trainer.model.forward(data, lens, hidden)
+                loss = trainer.criterion(decoded.view(-1, len(vocab['char'])), target)
+                total_loss += loss.data.item()
+                loss.backward()
+
+                torch.nn.utils.clip_grad_norm_(trainer.params, args['max_grad_norm'])
+                trainer.optimizer.step()
+
+                hidden = repackage_hidden(hidden)
+
+                if (iteration + 1) % args['report_steps'] == 0:
+                    cur_loss = total_loss / args['report_steps']
+                    elapsed = time.time() - start_time
+                    logger.info(
+                        "| epoch {:5d} | {:5d}/{:5d} batches | sec/batch {:.6f} | loss {:5.2f} | ppl {:8.2f}".format(
+                            trainer.epoch,
+                            iteration + 1,
+                            total_batches,
+                            elapsed / args['report_steps'],
+                            cur_loss,
+                            math.exp(cur_loss),
+                        )
+                    )
+                    if args['wandb']:
+                        wandb.log({'train_loss': cur_loss}, step=trainer.global_step)
+                    total_loss = 0.0
+
+                iteration += 1
+                i += seq_len
+
+                # evaluate if necessary
+                if eval_within_epoch and trainer.global_step % args['eval_steps'] == 0:
+                    _, ppl, best_loss = evaluate_and_save(args, vocab, dev_data, trainer, best_loss, model_file, checkpoint_file, writer)
+                    if args['wandb']:
+                        wandb.log({'ppl': ppl, 'best_loss': best_loss, 'lr': get_current_lr(trainer, args)}, step=trainer.global_step)
+
+        # if eval_interval isn't provided, run evaluation after each epoch
+        if not eval_within_epoch or trainer.epoch == args['epochs']:
+            _, ppl, best_loss = evaluate_and_save(args, vocab, dev_data, trainer, best_loss, model_file, checkpoint_file, writer)
+            if args['wandb']:
+                wandb.log({'ppl': ppl, 'best_loss': best_loss, 'lr': get_current_lr(trainer, args)}, step=trainer.global_step)
+
+    if writer:
+        writer.close()
+    if args['wandb']:
+        wandb.finish()
+    return
+
+def evaluate(args):
+    model_file = build_model_filename(args)
+
+    model = CharacterLanguageModel.load(model_file).to(args['device'])
+    vocab = model.vocab
+    data = load_data(args['eval_file'], vocab, args['direction'])
+    criterion = torch.nn.CrossEntropyLoss()
+    
+    loss = evaluate_epoch(args, vocab, data, model, criterion)
+    logger.info(
+        "| best model | loss {:5.2f} | ppl {:8.2f}".format(
+            loss,
+            math.exp(loss),
+        )
+    )
+    return
+
+if __name__ == '__main__':
+    main()

stanza/stanza/models/identity_lemmatizer.py

@@ -0,0 +1,66 @@
+"""
+An identity lemmatizer that mimics the behavior of a normal lemmatizer but directly uses word as lemma.
+"""
+
+import os
+import argparse
+import logging
+import random
+
+from stanza.models.lemma.data import DataLoader
+from stanza.models.lemma import scorer
+from stanza.models.common import utils
+from stanza.models.common.doc import *
+from stanza.utils.conll import CoNLL
+from stanza.models import _training_logging
+
+logger = logging.getLogger('stanza')
+
+def parse_args(args=None):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--data_dir', type=str, default='data/lemma', help='Directory for all lemma data.')
+    parser.add_argument('--train_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--eval_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--output_file', type=str, default=None, help='Output CoNLL-U file.')
+    parser.add_argument('--gold_file', type=str, default=None, help='Output CoNLL-U file.')
+
+    parser.add_argument('--mode', default='train', choices=['train', 'predict'])
+    parser.add_argument('--shorthand', type=str, help='Shorthand')
+
+    parser.add_argument('--batch_size', type=int, default=50)
+    parser.add_argument('--seed', type=int, default=1234)
+
+    args = parser.parse_args(args=args)
+    return args
+
+def main(args=None):
+    args = parse_args(args=args)
+
+    random.seed(args.seed)
+
+    args = vars(args)
+
+    logger.info("[Launching identity lemmatizer...]")
+
+    if args['mode'] == 'train':
+        logger.info("[No training is required; will only generate evaluation output...]")
+    
+    document = CoNLL.conll2doc(input_file=args['eval_file'])
+    batch = DataLoader(document, args['batch_size'], args, evaluation=True, conll_only=True)
+    system_pred_file = args['output_file']
+    gold_file = args['gold_file']
+
+    # use identity mapping for prediction
+    preds = batch.doc.get([TEXT])
+
+    # write to file and score
+    batch.doc.set([LEMMA], preds)
+    CoNLL.write_doc2conll(batch.doc, system_pred_file)
+    if gold_file is not None:
+        _, _, score = scorer.score(system_pred_file, gold_file)
+
+        logger.info("Lemma score:")
+        logger.info("{} {:.2f}".format(args['shorthand'], score*100))
+
+if __name__ == '__main__':
+    main()

stanza/stanza/models/lang_identifier.py

@@ -0,0 +1,236 @@
+"""
+Entry point for training and evaluating a Bi-LSTM language identifier
+"""
+
+import argparse
+import json
+import logging
+import os
+import random
+import torch
+
+from datetime import datetime
+from stanza.models.common import utils
+from stanza.models.langid.data import DataLoader
+from stanza.models.langid.trainer import Trainer
+from stanza.utils.get_tqdm import get_tqdm
+
+tqdm = get_tqdm()
+
+logger = logging.getLogger('stanza')
+
+def parse_args(args=None):
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--batch_mode", help="custom settings when running in batch mode", action="store_true")
+    parser.add_argument("--batch_size", help="batch size for training", type=int, default=64)
+    parser.add_argument("--eval_length", help="length of strings to eval on", type=int, default=None)
+    parser.add_argument("--eval_set", help="eval on dev or test", default="test")
+    parser.add_argument("--data_dir", help="directory with train/dev/test data", default=None)
+    parser.add_argument("--load_name", help="path to load model from", default=None)
+    parser.add_argument("--mode", help="train or eval", default="train")
+    parser.add_argument("--num_epochs", help="number of epochs for training", type=int, default=50)
+    parser.add_argument("--randomize", help="take random substrings of samples", action="store_true")
+    parser.add_argument("--randomize_lengths_range", help="range of lengths to use when random sampling text",
+                        type=randomize_lengths_range, default="5,20")
+    parser.add_argument("--merge_labels_for_eval",
+                        help="merge some language labels for eval (e.g. \"zh-hans\" and \"zh-hant\" to \"zh\")", 
+                        action="store_true")
+    parser.add_argument("--save_best_epochs", help="save model for every epoch with new best score", action="store_true")
+    parser.add_argument("--save_name", help="where to save model", default=None)
+    utils.add_device_args(parser)
+    args = parser.parse_args(args=args)
+    return args
+
+
+def randomize_lengths_range(range_list):
+    """
+    Range of lengths for random samples
+    """
+    range_boundaries = [int(x) for x in range_list.split(",")]
+    assert range_boundaries[0] < range_boundaries[1], f"Invalid range: ({range_boundaries[0]}, {range_boundaries[1]})"
+    return range_boundaries
+
+
+def main(args=None):
+    args = parse_args(args=args)
+    torch.manual_seed(0)
+    if args.mode == "train":
+        train_model(args)
+    else:
+        eval_model(args)
+
+
+def build_indexes(args):
+    tag_to_idx = {}
+    char_to_idx = {}
+    train_files = [f"{args.data_dir}/{x}" for x in os.listdir(args.data_dir) if "train" in x]
+    for train_file in train_files:
+        with open(train_file) as curr_file:
+            lines = curr_file.read().strip().split("\n")
+        examples = [json.loads(line) for line in lines if line.strip()]
+        for example in examples:
+            label = example["label"]
+            if label not in tag_to_idx:
+                tag_to_idx[label] = len(tag_to_idx)
+            sequence = example["text"]
+            for char in list(sequence):
+                if char not in char_to_idx:
+                    char_to_idx[char] = len(char_to_idx)
+    char_to_idx["UNK"] = len(char_to_idx)
+    char_to_idx["<PAD>"] = len(char_to_idx)
+
+    return tag_to_idx, char_to_idx
+
+
+def train_model(args):
+    # set up indexes
+    tag_to_idx, char_to_idx = build_indexes(args)
+    # load training data
+    train_data = DataLoader(args.device)
+    train_files = [f"{args.data_dir}/{x}" for x in os.listdir(args.data_dir) if "train" in x]
+    train_data.load_data(args.batch_size, train_files, char_to_idx, tag_to_idx, args.randomize)
+    # load dev data
+    dev_data = DataLoader(args.device)
+    dev_files = [f"{args.data_dir}/{x}" for x in os.listdir(args.data_dir) if "dev" in x]
+    dev_data.load_data(args.batch_size, dev_files, char_to_idx, tag_to_idx, randomize=False, 
+                       max_length=args.eval_length)
+    # set up trainer
+    trainer_config = {
+        "model_path": args.save_name,
+        "char_to_idx": char_to_idx,
+        "tag_to_idx": tag_to_idx,
+        "batch_size": args.batch_size,
+        "lang_weights": train_data.lang_weights
+    }
+    if args.load_name:
+        trainer_config["load_name"] = args.load_name
+        logger.info(f"{datetime.now()}\tLoading model from: {args.load_name}")
+    trainer = Trainer(trainer_config, load_model=args.load_name is not None, device=args.device)
+    # run training
+    best_accuracy = 0.0
+    for epoch in range(1, args.num_epochs+1):
+        logger.info(f"{datetime.now()}\tEpoch {epoch}")
+        logger.info(f"{datetime.now()}\tNum training batches: {len(train_data.batches)}")
+
+        batches = train_data.batches
+        if not args.batch_mode:
+            batches = tqdm(batches)
+        for train_batch in batches:
+            inputs = (train_batch["sentences"], train_batch["targets"])
+            trainer.update(inputs)
+
+        logger.info(f"{datetime.now()}\tEpoch complete. Evaluating on dev data.")
+        curr_dev_accuracy, curr_confusion_matrix, curr_precisions, curr_recalls, curr_f1s = \
+            eval_trainer(trainer, dev_data, batch_mode=args.batch_mode)
+        logger.info(f"{datetime.now()}\tCurrent dev accuracy: {curr_dev_accuracy}")
+        if curr_dev_accuracy > best_accuracy:
+            logger.info(f"{datetime.now()}\tNew best score. Saving model.")
+            model_label = f"epoch{epoch}" if args.save_best_epochs else None
+            trainer.save(label=model_label)
+            with open(score_log_path(args.save_name), "w") as score_log_file:
+                for score_log in [{"dev_accuracy": curr_dev_accuracy}, curr_confusion_matrix, curr_precisions,
+                                  curr_recalls, curr_f1s]:
+                    score_log_file.write(json.dumps(score_log) + "\n")
+            best_accuracy = curr_dev_accuracy
+
+        # reload training data
+        logger.info(f"{datetime.now()}\tResampling training data.")
+        train_data.load_data(args.batch_size, train_files, char_to_idx, tag_to_idx, args.randomize)
+
+
+def score_log_path(file_path):
+    """
+    Helper that will determine corresponding log file (e.g. /path/to/demo.pt to /path/to/demo.json
+    """
+    model_suffix = os.path.splitext(file_path)
+    if model_suffix[1]:
+        score_log_path = f"{file_path[:-len(model_suffix[1])]}.json"
+    else:
+        score_log_path = f"{file_path}.json"
+    return score_log_path
+
+
+def eval_model(args):
+    # set up trainer
+    trainer_config = {
+        "model_path": None,
+        "load_name": args.load_name,
+        "batch_size": args.batch_size
+    }
+    trainer = Trainer(trainer_config, load_model=True, device=args.device)
+    # load test data
+    test_data = DataLoader(args.device)
+    test_files = [f"{args.data_dir}/{x}" for x in os.listdir(args.data_dir) if args.eval_set in x]
+    test_data.load_data(args.batch_size, test_files, trainer.model.char_to_idx, trainer.model.tag_to_idx, 
+                        randomize=False, max_length=args.eval_length)
+    curr_accuracy, curr_confusion_matrix, curr_precisions, curr_recalls, curr_f1s = \
+        eval_trainer(trainer, test_data, batch_mode=args.batch_mode, fine_grained=not args.merge_labels_for_eval)
+    logger.info(f"{datetime.now()}\t{args.eval_set} accuracy: {curr_accuracy}")
+    eval_save_path = args.save_name if args.save_name else score_log_path(args.load_name)
+    if not os.path.exists(eval_save_path) or args.save_name:
+        with open(eval_save_path, "w") as score_log_file:
+            for score_log in [{"dev_accuracy": curr_accuracy}, curr_confusion_matrix, curr_precisions,
+                              curr_recalls, curr_f1s]:
+                score_log_file.write(json.dumps(score_log) + "\n")
+        
+
+
+def eval_trainer(trainer, dev_data, batch_mode=False, fine_grained=True):
+    """
+    Produce dev accuracy and confusion matrix for a trainer
+    """
+
+    # set up confusion matrix
+    tag_to_idx = dev_data.tag_to_idx
+    idx_to_tag = dev_data.idx_to_tag
+    confusion_matrix = {}
+    for row_label in tag_to_idx:
+        confusion_matrix[row_label] = {}
+        for col_label in tag_to_idx:
+            confusion_matrix[row_label][col_label] = 0
+
+    # process dev batches
+    batches = dev_data.batches
+    if not batch_mode:
+        batches = tqdm(batches)
+    for dev_batch in batches:
+        inputs = (dev_batch["sentences"], dev_batch["targets"])
+        predictions = trainer.predict(inputs)
+        for target_idx, prediction in zip(dev_batch["targets"], predictions):
+            prediction_label = idx_to_tag[prediction] if fine_grained else idx_to_tag[prediction].split("-")[0]
+            confusion_matrix[idx_to_tag[target_idx]][prediction_label] += 1
+
+    # calculate dev accuracy
+    total_examples = sum([sum([confusion_matrix[i][j] for j in confusion_matrix[i]]) for i in confusion_matrix])
+    total_correct = sum([confusion_matrix[i][i] for i in confusion_matrix])
+    dev_accuracy = float(total_correct) / float(total_examples)
+
+    # calculate precision, recall, F1
+    precision_scores = {"type": "precision"}
+    recall_scores = {"type": "recall"}
+    f1_scores = {"type": "f1"}
+    for prediction_label in tag_to_idx:
+        total = sum([confusion_matrix[k][prediction_label] for k in tag_to_idx])
+        if total != 0.0:
+            precision_scores[prediction_label] = float(confusion_matrix[prediction_label][prediction_label])/float(total)
+        else:
+            precision_scores[prediction_label] = 0.0
+    for target_label in tag_to_idx:
+        total = sum([confusion_matrix[target_label][k] for k in tag_to_idx])
+        if total != 0:
+            recall_scores[target_label] = float(confusion_matrix[target_label][target_label])/float(total)
+        else:
+            recall_scores[target_label] = 0.0
+    for label in tag_to_idx:
+        if precision_scores[label] == 0.0 and recall_scores[label] == 0.0:
+            f1_scores[label] = 0.0
+        else:
+            f1_scores[label] = \
+                2.0 * (precision_scores[label] * recall_scores[label]) / (precision_scores[label] + recall_scores[label])
+
+    return dev_accuracy, confusion_matrix, precision_scores, recall_scores, f1_scores
+
+
+if __name__ == "__main__":
+    main()
+

stanza/stanza/models/mwt_expander.py

@@ -0,0 +1,322 @@
+"""
+Entry point for training and evaluating a multi-word token (MWT) expander.
+
+This MWT expander combines a neural sequence-to-sequence architecture with a dictionary
+to decode the token into multiple words.
+For details please refer to paper: https://nlp.stanford.edu/pubs/qi2018universal.pdf
+
+In the case of a dataset where all of the MWT exactly split into the words
+composing the MWT, a classifier over the characters is used instead of the seq2seq
+"""
+
+import sys
+import os
+import shutil
+import time
+from datetime import datetime
+import argparse
+import logging
+import math
+import numpy as np
+import random
+import torch
+from torch import nn, optim
+import copy
+
+from stanza.models.mwt.data import DataLoader, BinaryDataLoader
+from stanza.models.mwt.utils import mwts_composed_of_words
+from stanza.models.mwt.vocab import Vocab
+from stanza.models.mwt.trainer import Trainer
+from stanza.models.mwt import scorer
+from stanza.models.common import utils
+import stanza.models.common.seq2seq_constant as constant
+from stanza.models.common.doc import Document
+from stanza.utils.conll import CoNLL
+from stanza.models import _training_logging
+
+logger = logging.getLogger('stanza')
+
+def build_argparse():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--data_dir', type=str, default='data/mwt', help='Root dir for saving models.')
+    parser.add_argument('--train_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--eval_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--output_file', type=str, default=None, help='Output CoNLL-U file.')
+    parser.add_argument('--gold_file', type=str, default=None, help='Output CoNLL-U file.')
+
+    parser.add_argument('--mode', default='train', choices=['train', 'predict'])
+    parser.add_argument('--lang', type=str, help='Language')
+    parser.add_argument('--shorthand', type=str, help="Treebank shorthand")
+
+    parser.add_argument('--no_dict', dest='ensemble_dict', action='store_false', help='Do not ensemble dictionary with seq2seq. By default ensemble a dict.')
+    parser.add_argument('--ensemble_early_stop', action='store_true', help='Early stopping based on ensemble performance.')
+    parser.add_argument('--dict_only', action='store_true', help='Only train a dictionary-based MWT expander.')
+
+    parser.add_argument('--hidden_dim', type=int, default=100)
+    parser.add_argument('--emb_dim', type=int, default=50)
+    parser.add_argument('--num_layers', type=int, default=None, help='Number of layers in model encoder.  Defaults to 1 for seq2seq, 2 for classifier')
+    parser.add_argument('--emb_dropout', type=float, default=0.5)
+    parser.add_argument('--dropout', type=float, default=0.5)
+    parser.add_argument('--max_dec_len', type=int, default=50)
+    parser.add_argument('--beam_size', type=int, default=1)
+    parser.add_argument('--attn_type', default='soft', choices=['soft', 'mlp', 'linear', 'deep'], help='Attention type')
+    parser.add_argument('--no_copy', dest='copy', action='store_false', help='Do not use copy mechanism in MWT expansion. By default copy mechanism is used to improve generalization.')
+
+    parser.add_argument('--augment_apos', default=0.01, type=float, help='At training time, how much to augment |\'| to |"| |’| |ʼ|')
+    parser.add_argument('--force_exact_pieces', default=None, action='store_true', help='If possible, make the text of the pieces of the MWT add up to the token itself.  (By default, this is determined from the dataset.)')
+    parser.add_argument('--no_force_exact_pieces', dest='force_exact_pieces', action='store_false', help="Don't make the text of the pieces of the MWT add up to the token itself.  (By default, this is determined from the dataset.)")
+
+    parser.add_argument('--sample_train', type=float, default=1.0, help='Subsample training data.')
+    parser.add_argument('--optim', type=str, default='adam', help='sgd, adagrad, adam or adamax.')
+    parser.add_argument('--lr', type=float, default=1e-3, help='Learning rate')
+    parser.add_argument('--lr_decay', type=float, default=0.9)
+    parser.add_argument('--decay_epoch', type=int, default=30, help="Decay the lr starting from this epoch.")
+    parser.add_argument('--num_epoch', type=int, default=30)
+    parser.add_argument('--batch_size', type=int, default=50)
+    parser.add_argument('--max_grad_norm', type=float, default=5.0, help='Gradient clipping.')
+    parser.add_argument('--log_step', type=int, default=20, help='Print log every k steps.')
+    parser.add_argument('--save_dir', type=str, default='saved_models/mwt', help='Root dir for saving models.')
+    parser.add_argument('--save_name', type=str, default=None, help="File name to save the model")
+    parser.add_argument('--save_each_name', type=str, default=None, help="Save each model in sequence to this pattern.  Mostly for testing")
+
+    parser.add_argument('--seed', type=int, default=1234)
+    utils.add_device_args(parser)
+
+    parser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name')
+    parser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+    return parser
+
+def parse_args(args=None):
+    parser = build_argparse()
+    args = parser.parse_args(args=args)
+
+    if args.wandb_name:
+        args.wandb = True
+
+    return args
+
+def main(args=None):
+    args = parse_args(args=args)
+
+    utils.set_random_seed(args.seed)
+
+    args = vars(args)
+    logger.info("Running MWT expander in {} mode".format(args['mode']))
+
+    if args['mode'] == 'train':
+        train(args)
+    else:
+        evaluate(args)
+
+def train(args):
+    # load data
+    logger.debug('max_dec_len: %d' % args['max_dec_len'])
+    logger.debug("Loading data with batch size {}...".format(args['batch_size']))
+    train_doc = CoNLL.conll2doc(input_file=args['train_file'])
+    train_batch = DataLoader(train_doc, args['batch_size'], args, evaluation=False)
+    vocab = train_batch.vocab
+    args['vocab_size'] = vocab.size
+    dev_doc = CoNLL.conll2doc(input_file=args['eval_file'])
+    dev_batch = DataLoader(dev_doc, args['batch_size'], args, vocab=vocab, evaluation=True)
+
+    utils.ensure_dir(args['save_dir'])
+    save_name = args['save_name'] if args['save_name'] else '{}_mwt_expander.pt'.format(args['shorthand'])
+    model_file = os.path.join(args['save_dir'], save_name)
+
+    save_each_name = None
+    if args['save_each_name']:
+        save_each_name = os.path.join(args['save_dir'], args['save_each_name'])
+        save_each_name = utils.build_save_each_filename(save_each_name)
+
+    # pred and gold path
+    system_pred_file = args['output_file']
+    gold_file = args['gold_file']
+
+    # skip training if the language does not have training or dev data
+    if len(train_batch) == 0:
+        logger.warning("Skip training because no data available...")
+        return
+    dev_mwt = dev_doc.get_mwt_expansions(False)
+    if len(dev_batch) == 0 and args.get('dict_only', False):
+        logger.warning("Training data available, but dev data has no MWTs.  Only training a dict based MWT")
+        args['dict_only'] = True
+
+    if args['force_exact_pieces'] and not mwts_composed_of_words(train_doc):
+        raise ValueError("Cannot train model with --force_exact_pieces, as the MWT in this dataset are not entirely composed of their subwords")
+
+    if args['force_exact_pieces'] is None and mwts_composed_of_words(train_doc):
+        # the force_exact_pieces mechanism trains a separate version of the MWT expander in the Trainer
+        # (the training loop here does not need to change)
+        # in this model, a classifier distinguishes whether or not a location is a split
+        # and the text is copied exactly from the input rather than created via seq2seq
+        # this behavior can be turned off at training time with --no_force_exact_pieces
+        logger.info("Train MWTs entirely composed of their subwords.  Training the MWT to match that paradigm as closely as possible")
+        args['force_exact_pieces'] = True
+
+    if args['force_exact_pieces']:
+        logger.info("Reconverting to BinaryDataLoader")
+        train_batch = BinaryDataLoader(train_doc, args['batch_size'], args, evaluation=False)
+        vocab = train_batch.vocab
+        args['vocab_size'] = vocab.size
+        dev_batch = BinaryDataLoader(dev_doc, args['batch_size'], args, vocab=vocab, evaluation=True)
+
+    if args['num_layers'] is None:
+        if args['force_exact_pieces']:
+            args['num_layers'] = 2
+        else:
+            args['num_layers'] = 1
+
+    # train a dictionary-based MWT expander
+    trainer = Trainer(args=args, vocab=vocab, device=args['device'])
+    logger.info("Training dictionary-based MWT expander...")
+    trainer.train_dict(train_batch.doc.get_mwt_expansions(evaluation=False))
+    logger.info("Evaluating on dev set...")
+    dev_preds = trainer.predict_dict(dev_batch.doc.get_mwt_expansions(evaluation=True))
+    doc = copy.deepcopy(dev_batch.doc)
+    doc.set_mwt_expansions(dev_preds, fake_dependencies=True)
+    CoNLL.write_doc2conll(doc, system_pred_file)
+    _, _, dev_f = scorer.score(system_pred_file, gold_file)
+    logger.info("Dev F1 = {:.2f}".format(dev_f * 100))
+
+    if args.get('dict_only', False):
+        # save dictionaries
+        trainer.save(model_file)
+    else:
+        # train a seq2seq model
+        logger.info("Training seq2seq-based MWT expander...")
+        global_step = 0
+        steps_per_epoch = math.ceil(len(train_batch) / args['batch_size'])
+        max_steps = steps_per_epoch * args['num_epoch']
+        dev_score_history = []
+        best_dev_preds = []
+        current_lr = args['lr']
+        global_start_time = time.time()
+        format_str = '{}: step {}/{} (epoch {}/{}), loss = {:.6f} ({:.3f} sec/batch), lr: {:.6f}'
+
+        if args['wandb']:
+            import wandb
+            wandb_name = args['wandb_name'] if args['wandb_name'] else "%s_mwt" % args['shorthand']
+            wandb.init(name=wandb_name, config=args)
+            wandb.run.define_metric('train_loss', summary='min')
+            wandb.run.define_metric('dev_score', summary='max')
+
+        # start training
+        for epoch in range(1, args['num_epoch']+1):
+            train_loss = 0
+            for i, batch in enumerate(train_batch.to_loader()):
+                start_time = time.time()
+                global_step += 1
+                loss = trainer.update(batch, eval=False) # update step
+                train_loss += loss
+                if global_step % args['log_step'] == 0:
+                    duration = time.time() - start_time
+                    logger.info(format_str.format(datetime.now().strftime("%Y-%m-%d %H:%M:%S"), global_step,\
+                                                  max_steps, epoch, args['num_epoch'], loss, duration, current_lr))
+
+            if save_each_name:
+                trainer.save(save_each_name % epoch)
+                logger.info("Saved epoch %d model to %s" % (epoch, save_each_name % epoch))
+
+            # eval on dev
+            logger.info("Evaluating on dev set...")
+            dev_preds = []
+            for i, batch in enumerate(dev_batch.to_loader()):
+                preds = trainer.predict(batch)
+                dev_preds += preds
+            if args.get('ensemble_dict', False) and args.get('ensemble_early_stop', False):
+                logger.info("[Ensembling dict with seq2seq model...]")
+                dev_preds = trainer.ensemble(dev_batch.doc.get_mwt_expansions(evaluation=True), dev_preds)
+            doc = copy.deepcopy(dev_batch.doc)
+            doc.set_mwt_expansions(dev_preds, fake_dependencies=True)
+            CoNLL.write_doc2conll(doc, system_pred_file)
+            _, _, dev_score = scorer.score(system_pred_file, gold_file)
+            train_loss = train_loss / train_batch.num_examples * args['batch_size'] # avg loss per batch
+            logger.info("epoch {}: train_loss = {:.6f}, dev_score = {:.4f}".format(epoch, train_loss, dev_score))
+
+            if args['wandb']:
+                wandb.log({'train_loss': train_loss, 'dev_score': dev_score})
+
+            # save best model
+            if epoch == 1 or dev_score > max(dev_score_history):
+                trainer.save(model_file)
+                logger.info("new best model saved.")
+                best_dev_preds = dev_preds
+
+            # lr schedule
+            if epoch > args['decay_epoch'] and dev_score <= dev_score_history[-1]:
+                current_lr *= args['lr_decay']
+                trainer.change_lr(current_lr)
+
+            dev_score_history += [dev_score]
+
+        logger.info("Training ended with {} epochs.".format(epoch))
+
+        if args['wandb']:
+            wandb.finish()
+
+        best_f, best_epoch = max(dev_score_history)*100, np.argmax(dev_score_history)+1
+        logger.info("Best dev F1 = {:.2f}, at epoch = {}".format(best_f, best_epoch))
+
+        # try ensembling with dict if necessary
+        if args.get('ensemble_dict', False):
+            logger.info("[Ensembling dict with seq2seq model...]")
+            dev_preds = trainer.ensemble(dev_batch.doc.get_mwt_expansions(evaluation=True), best_dev_preds)
+            doc = copy.deepcopy(dev_batch.doc)
+            doc.set_mwt_expansions(dev_preds, fake_dependencies=True)
+            CoNLL.write_doc2conll(doc, system_pred_file)
+            _, _, dev_score = scorer.score(system_pred_file, gold_file)
+            logger.info("Ensemble dev F1 = {:.2f}".format(dev_score*100))
+            best_f = max(best_f, dev_score)
+
+def evaluate(args):
+    # file paths
+    system_pred_file = args['output_file']
+    gold_file = args['gold_file']
+    model_file = args['save_name'] if args['save_name'] else '{}_mwt_expander.pt'.format(args['shorthand'])
+    if args['save_dir'] and not model_file.startswith(args['save_dir']) and not os.path.exists(model_file):
+        model_file = os.path.join(args['save_dir'], model_file)
+
+    # load model
+    trainer = Trainer(model_file=model_file, device=args['device'])
+    loaded_args, vocab = trainer.args, trainer.vocab
+
+    for k in args:
+        if k.endswith('_dir') or k.endswith('_file') or k in ['shorthand']:
+            loaded_args[k] = args[k]
+    logger.debug('max_dec_len: %d' % loaded_args['max_dec_len'])
+
+    # load data
+    logger.debug("Loading data with batch size {}...".format(args['batch_size']))
+    doc = CoNLL.conll2doc(input_file=args['eval_file'])
+    batch = DataLoader(doc, args['batch_size'], loaded_args, vocab=vocab, evaluation=True)
+
+    if len(batch) > 0:
+        dict_preds = trainer.predict_dict(batch.doc.get_mwt_expansions(evaluation=True))
+        # decide trainer type and run eval
+        if loaded_args['dict_only']:
+            preds = dict_preds
+        else:
+            logger.info("Running the seq2seq model...")
+            preds = []
+            for i, b in enumerate(batch.to_loader()):
+                preds += trainer.predict(b)
+
+            if loaded_args.get('ensemble_dict', False):
+                preds = trainer.ensemble(batch.doc.get_mwt_expansions(evaluation=True), preds)
+    else:
+        # skip eval if dev data does not exist
+        preds = []
+
+    # write to file and score
+    doc = copy.deepcopy(batch.doc)
+    doc.set_mwt_expansions(preds, fake_dependencies=True)
+    CoNLL.write_doc2conll(doc, system_pred_file)
+
+    if gold_file is not None:
+        _, _, score = scorer.score(system_pred_file, gold_file)
+
+        logger.info("MWT expansion score: {} {:.2f}".format(args['shorthand'], score*100))
+
+
+if __name__ == '__main__':
+    main()

stanza/stanza/models/ner_tagger.py

@@ -0,0 +1,492 @@
+"""
+Entry point for training and evaluating an NER tagger.
+
+This tagger uses BiLSTM layers with character and word-level representations, and a CRF decoding layer 
+to produce NER predictions.
+For details please refer to paper: https://nlp.stanford.edu/pubs/qi2018universal.pdf.
+"""
+
+import sys
+import os
+import time
+from datetime import datetime
+import argparse
+import logging
+import numpy as np
+import random
+import re
+import json
+import torch
+from torch import nn, optim
+
+from stanza.models.ner.data import DataLoader
+from stanza.models.ner.trainer import Trainer
+from stanza.models.ner import scorer
+from stanza.models.common import utils
+from stanza.models.common.pretrain import Pretrain
+from stanza.utils.conll import CoNLL
+from stanza.models.common.doc import *
+from stanza.models import _training_logging
+
+from stanza.models.common.peft_config import add_peft_args, resolve_peft_args
+from stanza.utils.confusion import confusion_to_weighted_f1, format_confusion
+
+logger = logging.getLogger('stanza')
+
+def build_argparse():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--data_dir', type=str, default='data/ner', help='Directory of NER data.')
+    parser.add_argument('--wordvec_dir', type=str, default='extern_data/word2vec', help='Directory of word vectors')
+    parser.add_argument('--wordvec_file', type=str, default='', help='File that contains word vectors')
+    parser.add_argument('--wordvec_pretrain_file', type=str, default=None, help='Exact name of the pretrain file to read')
+    parser.add_argument('--train_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--eval_file', type=str, default=None, help='Input file for data loader.')
+    parser.add_argument('--eval_output_file', type=str, default=None, help='Where to write results: text, gold, pred.  If None, no results file printed')
+
+    parser.add_argument('--mode', default='train', choices=['train', 'predict'])
+    parser.add_argument('--finetune', action='store_true', help='Load existing model during `train` mode from `save_dir` path')
+    parser.add_argument('--finetune_load_name', type=str, default=None, help='Model to load when finetuning')
+    parser.add_argument('--train_classifier_only', action='store_true',
+                        help='In case of applying Transfer-learning approach and training only the classifier layer this will freeze gradient propagation for all other layers.')
+    parser.add_argument('--shorthand', type=str, help="Treebank shorthand")
+
+    parser.add_argument('--hidden_dim', type=int, default=256)
+    parser.add_argument('--char_hidden_dim', type=int, default=100)
+    parser.add_argument('--word_emb_dim', type=int, default=100)
+    parser.add_argument('--char_emb_dim', type=int, default=100)
+    parser.add_argument('--num_layers', type=int, default=1)
+    parser.add_argument('--char_num_layers', type=int, default=1)
+    parser.add_argument('--pretrain_max_vocab', type=int, default=100000)
+    parser.add_argument('--word_dropout', type=float, default=0.01, help="How often to remove a word at training time.  Set to a small value to train unk when finetuning word embeddings")
+    parser.add_argument('--locked_dropout', type=float, default=0.0)
+    parser.add_argument('--dropout', type=float, default=0.5)
+    parser.add_argument('--rec_dropout', type=float, default=0, help="Word recurrent dropout")
+    parser.add_argument('--char_rec_dropout', type=float, default=0, help="Character recurrent dropout")
+    parser.add_argument('--char_dropout', type=float, default=0, help="Character-level language model dropout")
+    parser.add_argument('--no_char', dest='char', action='store_false', help="Turn off training a character model.")
+    parser.add_argument('--charlm', action='store_true', help="Turn on contextualized char embedding using pretrained character-level language model.")
+    parser.add_argument('--charlm_save_dir', type=str, default='saved_models/charlm', help="Root dir for pretrained character-level language model.")
+    parser.add_argument('--charlm_shorthand', type=str, default=None, help="Shorthand for character-level language model training corpus.")
+    parser.add_argument('--charlm_forward_file', type=str, default=None, help="Exact path to use for forward charlm")
+    parser.add_argument('--charlm_backward_file', type=str, default=None, help="Exact path to use for backward charlm")
+    parser.add_argument('--char_lowercase', dest='char_lowercase', action='store_true', help="Use lowercased characters in character model.")
+    parser.add_argument('--no_lowercase', dest='lowercase', action='store_false', help="Use cased word vectors.")
+    parser.add_argument('--no_emb_finetune', dest='emb_finetune', action='store_false', help="Turn off finetuning of the embedding matrix.")
+    parser.add_argument('--emb_finetune_known_only', dest='emb_finetune_known_only', action='store_true', help="Finetune the embedding matrix only for words in the embedding.  (Default: finetune words not in the embedding as well)  This may be useful for very large datasets where obscure words are only trained once in a while, such as French-WikiNER")
+    parser.add_argument('--no_input_transform', dest='input_transform', action='store_false', help="Do not use input transformation layer before tagger lstm.")
+    parser.add_argument('--scheme', type=str, default='bioes', help="The tagging scheme to use: bio or bioes.")
+    parser.add_argument('--train_scheme', type=str, default=None, help="The tagging scheme to use when training: bio or bioes.  Overrides --scheme for the training set")
+
+    parser.add_argument('--bert_model', type=str, default=None, help="Use an external bert model (requires the transformers package)")
+    parser.add_argument('--no_bert_model', dest='bert_model', action="store_const", const=None, help="Don't use bert")
+    parser.add_argument('--bert_hidden_layers', type=int, default=None, help="How many layers of hidden state to use from the transformer")
+    parser.add_argument('--bert_finetune', default=False, action='store_true', help='Finetune the bert (or other transformer)')
+    parser.add_argument('--gradient_checkpointing', default=False, action='store_true', help='Checkpoint intermediate gradients between layers to save memory at the cost of training steps')
+    parser.add_argument('--no_bert_finetune', dest='bert_finetune', action='store_false', help="Don't finetune the bert (or other transformer)")
+    parser.add_argument('--bert_learning_rate', default=1.0, type=float, help='Scale the learning rate for transformer finetuning by this much')
+    parser.add_argument('--second_optim', type=str, default=None, help='once first optimizer converged, tune the model again. with: sgd, adagrad, adam or adamax.')
+    parser.add_argument('--second_bert_learning_rate', default=0, type=float, help='Secondary stage transformer finetuning learning rate scale')
+
+    parser.add_argument('--sample_train', type=float, default=1.0, help='Subsample training data.')
+    parser.add_argument('--optim', type=str, default='sgd', help='sgd, adagrad, adam or adamax.')
+    parser.add_argument('--lr', type=float, default=0.1, help='Learning rate.')
+    parser.add_argument('--min_lr', type=float, default=1e-4, help='Minimum learning rate to stop training.')
+    parser.add_argument('--second_lr', type=float, default=5e-3, help='Secondary learning rate')
+    parser.add_argument('--momentum', type=float, default=0, help='Momentum for SGD.')
+    parser.add_argument('--lr_decay', type=float, default=0.5, help="LR decay rate.")
+    parser.add_argument('--patience', type=int, default=3, help="Patience for LR decay.")
+
+    parser.add_argument('--connect_output_layers', action='store_true', default=False, help='Connect one output layer to the input of the next output layer.  By default, those layers are all separate')
+    parser.add_argument('--predict_tagset', type=int, default=None, help='Which tagset to predict if there are multiple tagsets.  Will default to 0.  Default of None allows the model to remember the value from training time, but be overridden at test time')
+
+    parser.add_argument('--ignore_tag_scores', type=str, default=None, help="Which tags to ignore, if any, when scoring dev & test sets")
+
+    parser.add_argument('--max_steps', type=int, default=200000)
+    parser.add_argument('--max_steps_no_improve', type=int, default=2500, help='if the model doesn\'t improve after this many steps, give up or switch to new optimizer.')
+    parser.add_argument('--eval_interval', type=int, default=500)
+    parser.add_argument('--batch_size', type=int, default=32)
+    parser.add_argument('--max_batch_words', type=int, default=800, help='Long sentences can overwhelm even a large GPU when finetuning a transformer on otherwise reasonable batch sizes.  This cuts off those batches early')
+    parser.add_argument('--max_grad_norm', type=float, default=5.0, help='Gradient clipping.')
+    parser.add_argument('--log_step', type=int, default=20, help='Print log every k steps.')
+    parser.add_argument('--log_norms', action='store_true', default=False, help='Log the norms of all the parameters (noisy!)')
+    parser.add_argument('--save_dir', type=str, default='saved_models/ner', help='Root dir for saving models.')
+    parser.add_argument('--save_name', type=str, default="{shorthand}_{embedding}_{finetune}_nertagger.pt", help="File name to save the model")
+
+    parser.add_argument('--seed', type=int, default=1234)
+    utils.add_device_args(parser)
+
+    parser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name')
+    parser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+    return parser
+
+def parse_args(args=None):
+    parser = build_argparse()
+    add_peft_args(parser)
+    args = parser.parse_args(args=args)
+    resolve_peft_args(args, logger)
+
+    if args.wandb_name:
+        args.wandb = True
+
+    args = vars(args)
+    return args
+
+def main(args=None):
+    args = parse_args(args=args)
+
+    utils.set_random_seed(args['seed'])
+
+    logger.info("Running NER tagger in {} mode".format(args['mode']))
+
+    if args['mode'] == 'train':
+        return train(args)
+    else:
+        evaluate(args)
+
+def load_pretrain(args):
+    # load pretrained vectors
+    if args['wordvec_pretrain_file']:
+        pretrain_file = args['wordvec_pretrain_file']
+        pretrain = Pretrain(pretrain_file, None, args['pretrain_max_vocab'], save_to_file=False)
+    else:
+        if len(args['wordvec_file']) == 0:
+            vec_file = utils.get_wordvec_file(args['wordvec_dir'], args['shorthand'])
+        else:
+            vec_file = args['wordvec_file']
+        # do not save pretrained embeddings individually
+        pretrain = Pretrain(None, vec_file, args['pretrain_max_vocab'], save_to_file=False)
+    return pretrain
+
+def model_file_name(args):
+    return utils.standard_model_file_name(args, "nertagger")
+
+def get_known_tags(tags):
+    """
+    Tags are stored in the dataset as a list of list of tags
+
+    This returns a sorted list for each column of tags in the dataset
+    """
+    max_columns = max(len(word) for sent in tags for word in sent)
+    known_tags = [set() for _ in range(max_columns)]
+    for sent in tags:
+        for word in sent:
+            for tag_idx, tag in enumerate(word):
+                known_tags[tag_idx].add(tag)
+    return [sorted(x) for x in known_tags]
+
+def warn_missing_tags(tag_vocab, data_tags, error_msg, bioes_to_bio=False):
+    """
+    Check for tags missing from the tag_vocab.
+
+    Given a tag_vocab and the known tags in the format used by
+    ner.data, go through the tags in the dataset and look for any
+    which aren't in the tag_vocab.
+
+    error_msg is something like "training set" or "eval file" to
+    indicate where the missing tags came from.
+    """
+    tag_depth = max(max(len(tags) for tags in sentence) for sentence in data_tags)
+
+    if tag_depth != len(tag_vocab.lens()):
+        logger.warning("Test dataset has a different number of tag types compared to the model: %d vs %d", tag_depth, len(tag_vocab.lens()))
+    for tag_set_idx in range(min(tag_depth, len(tag_vocab.lens()))):
+        tag_set = tag_vocab.items(tag_set_idx)
+        if len(tag_vocab.lens()) > 1:
+            current_error_msg = error_msg + " tag set %d" % tag_set_idx
+        else:
+            current_error_msg = error_msg
+
+        current_tags = set([word[tag_set_idx] for sentence in data_tags for word in sentence])
+        if bioes_to_bio:
+            current_tags = set([re.sub("^E-", "I-", re.sub("^S-", "B-", x)) for x in current_tags])
+        utils.warn_missing_tags(tag_set, current_tags, current_error_msg)
+
+def train(args):
+    model_file = model_file_name(args)
+
+    save_dir, save_name = os.path.split(model_file)
+    utils.ensure_dir(save_dir)
+    if args['save_dir'] is None:
+        args['save_dir'] = save_dir
+    args['save_name'] = save_name
+
+    utils.log_training_args(args, logger)
+
+    pretrain = None
+    vocab = None
+    trainer = None
+
+    if args['finetune'] and args['finetune_load_name']:
+        logger.warning('Finetune is ON. Using model from "{}"'.format(args['finetune_load_name']))
+        _, trainer, vocab = load_model(args, args['finetune_load_name'])
+    elif args['finetune'] and os.path.exists(model_file):
+        logger.warning('Finetune is ON. Using model from "{}"'.format(model_file))
+        _, trainer, vocab = load_model(args, model_file)
+    else:
+        if args['finetune']:
+            raise FileNotFoundError('Finetune is set to true but model file is not found: {}'.format(model_file))
+
+        pretrain = load_pretrain(args)
+
+        if pretrain is not None:
+            word_emb_dim = pretrain.emb.shape[1]
+            if args['word_emb_dim'] and args['word_emb_dim'] != word_emb_dim:
+                logger.warning("Embedding file has a dimension of {}.  Model will be built with that size instead of {}".format(word_emb_dim, args['word_emb_dim']))
+            args['word_emb_dim'] = word_emb_dim
+
+        if args['charlm']:
+            if args['charlm_shorthand'] is None:
+                raise ValueError("CharLM Shorthand is required for loading pretrained CharLM model...")
+            logger.info('Using pretrained contextualized char embedding')
+            if not args['charlm_forward_file']:
+                args['charlm_forward_file'] = '{}/{}_forward_charlm.pt'.format(args['charlm_save_dir'], args['charlm_shorthand'])
+            if not args['charlm_backward_file']:
+                args['charlm_backward_file'] = '{}/{}_backward_charlm.pt'.format(args['charlm_save_dir'], args['charlm_shorthand'])
+
+    # load data
+    logger.info("Loading training data with batch size %d from %s", args['batch_size'], args['train_file'])
+    with open(args['train_file']) as fin:
+        train_doc = Document(json.load(fin))
+    logger.info("Loaded %d sentences of training data", len(train_doc.sentences))
+    if len(train_doc.sentences) == 0:
+        raise ValueError("File %s exists but has no usable training data" % args['train_file'])
+    train_batch = DataLoader(train_doc, args['batch_size'], args, pretrain, vocab=vocab, evaluation=False, scheme=args.get('train_scheme'), max_batch_words=args['max_batch_words'])
+    vocab = train_batch.vocab
+    logger.info("Loading dev data from %s", args['eval_file'])
+    with open(args['eval_file']) as fin:
+        dev_doc = Document(json.load(fin))
+    logger.info("Loaded %d sentences of dev data", len(dev_doc.sentences))
+    if len(dev_doc.sentences) == 0:
+        raise ValueError("File %s exists but has no usable dev data" % args['train_file'])
+    dev_batch = DataLoader(dev_doc, args['batch_size'], args, pretrain, vocab=vocab, evaluation=True)
+
+    train_tags = get_known_tags(train_batch.tags)
+    logger.info("Training data has %d columns of tags", len(train_tags))
+    for tag_idx, tags in enumerate(train_tags):
+        logger.info("Tags present in training set at column %d:\n  Tags without BIES markers: %s\n  Tags with B-, I-, E-, or S-: %s",
+                    tag_idx,
+                    " ".join(sorted(set(i for i in tags if i[:2] not in ('B-', 'I-', 'E-', 'S-')))),
+                    " ".join(sorted(set(i[2:] for i in tags if i[:2] in ('B-', 'I-', 'E-', 'S-')))))
+
+    # skip training if the language does not have training or dev data
+    if len(train_batch) == 0 or len(dev_batch) == 0:
+        logger.info("Skip training because no data available...")
+        return
+
+    logger.info("Training tagger...")
+    if trainer is None: # init if model was not loaded previously from file
+        trainer = Trainer(args=args, vocab=vocab, pretrain=pretrain, device=args['device'],
+                          train_classifier_only=args['train_classifier_only'])
+
+    if args['finetune']:
+        warn_missing_tags(trainer.vocab['tag'], train_batch.tags, "training set")
+    # the evaluation will coerce the tags to the proper scheme,
+    # so we won't need to alert for not having S- or E- tags
+    bioes_to_bio = args['train_scheme'] == 'bio' and args['scheme'] == 'bioes'
+    warn_missing_tags(trainer.vocab['tag'], dev_batch.tags, "dev set", bioes_to_bio=bioes_to_bio)
+
+    # TODO: might still want to add multiple layers of tag evaluation to the scorer
+    dev_gold_tags = [[x[trainer.args['predict_tagset']] for x in tags] for tags in dev_batch.tags]
+
+    logger.info(trainer.model)
+
+    global_step = 0
+    max_steps = args['max_steps']
+    dev_score_history = []
+    best_dev_preds = []
+    current_lr = trainer.optimizer.param_groups[0]['lr']
+    global_start_time = time.time()
+    format_str = '{}: step {}/{}, loss = {:.6f} ({:.3f} sec/batch), lr: {:.6f}'
+
+    # LR scheduling
+    if args['lr_decay'] > 0:
+        # learning rate changes on plateau -- no improvement on model for patience number of epochs
+        # change is made as a factor of the learning rate decay
+        scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(trainer.optimizer, mode='max', factor=args['lr_decay'],
+                                                               patience=args['patience'], verbose=True, min_lr=args['min_lr'])
+    else:
+        scheduler = None
+
+    if args['wandb']:
+        import wandb
+        wandb_name = args['wandb_name'] if args['wandb_name'] else "%s_ner" % args['shorthand']
+        wandb.init(name=wandb_name, config=args)
+        wandb.run.define_metric('train_loss', summary='min')
+        wandb.run.define_metric('dev_score', summary='max')
+        # track gradients!
+        wandb.watch(trainer.model, log_freq=4, log="gradients")
+
+    # start training
+    last_best_step = 0
+    train_loss = 0
+    is_second_optim = False
+    while True:
+        should_stop = False
+        for i, batch in enumerate(train_batch):
+            start_time = time.time()
+            global_step += 1
+            loss = trainer.update(batch, eval=False) # update step
+            train_loss += loss
+            if global_step % args['log_step'] == 0:
+                duration = time.time() - start_time
+                logger.info(format_str.format(datetime.now().strftime("%Y-%m-%d %H:%M:%S"), global_step,
+                                              max_steps, loss, duration, current_lr))
+            if global_step % args['eval_interval'] == 0:
+                # eval on dev
+                logger.info("Evaluating on dev set...")
+                dev_preds = []
+                for batch in dev_batch:
+                    preds = trainer.predict(batch)
+                    dev_preds += preds
+                _, _, dev_score, _ = scorer.score_by_entity(dev_preds, dev_gold_tags, ignore_tags=args['ignore_tag_scores'])
+
+                train_loss = train_loss / args['eval_interval'] # avg loss per batch
+                logger.info("step {}: train_loss = {:.6f}, dev_score = {:.4f}".format(global_step, train_loss, dev_score))
+                if args['wandb']:
+                    wandb.log({'train_loss': train_loss, 'dev_score': dev_score})
+                train_loss = 0
+
+                # save best model
+                if len(dev_score_history) == 0 or dev_score > max(dev_score_history):
+                    trainer.save(model_file)
+                    last_best_step = global_step
+                    logger.info("New best model saved.")
+                    best_dev_preds = dev_preds
+
+                dev_score_history += [dev_score]
+                logger.info("")
+
+                # lr schedule
+                if scheduler is not None:
+                    scheduler.step(dev_score)
+            
+                if args['log_norms']:
+                    trainer.model.log_norms()
+
+            # check stopping
+            current_lr = trainer.optimizer.param_groups[0]['lr']
+            if (global_step - last_best_step) >= args['max_steps_no_improve'] or global_step >= args['max_steps'] or current_lr <= args['min_lr']:
+                if (global_step - last_best_step) >= args['max_steps_no_improve']:
+                    logger.info("{} steps without improvement...".format((global_step - last_best_step)))
+                if not is_second_optim and args['second_optim'] is not None:
+                    logger.info("Switching to second optimizer: {}".format(args['second_optim']))
+                    logger.info('Reloading best model to continue from current local optimum')
+                    trainer = Trainer(args=args, vocab=vocab, pretrain=pretrain, device=args['device'],
+                                      train_classifier_only=args['train_classifier_only'], model_file=model_file, second_optim=True)
+                    is_second_optim = True
+                    last_best_step = global_step
+                    current_lr = trainer.optimizer.param_groups[0]['lr']
+                else:
+                    logger.info("stopping...")
+                    should_stop = True
+                    break
+
+        if should_stop:
+            break
+
+        train_batch.reshuffle()
+
+    logger.info("Training ended with {} steps.".format(global_step))
+
+    if args['wandb']:
+        wandb.finish()
+
+    if len(dev_score_history) > 0:
+        best_f, best_eval = max(dev_score_history)*100, np.argmax(dev_score_history)+1
+        logger.info("Best dev F1 = {:.2f}, at iteration = {}".format(best_f, best_eval * args['eval_interval']))
+    else:
+        logger.info("Dev set never evaluated.  Saving final model.")
+        trainer.save(model_file)
+
+    return trainer
+
+def write_ner_results(filename, batch, preds, predict_tagset):
+    if len(batch.tags) != len(preds):
+        raise ValueError("Unexpected batch vs pred lengths: %d vs %d" % (len(batch.tags), len(preds)))
+
+    with open(filename, "w", encoding="utf-8") as fout:
+        tag_idx = 0
+        for b in batch:
+            # b[0] is words, b[5] is orig_idx
+            # a namedtuple would make this cleaner without being much slower
+            text = utils.unsort(b[0], b[5])
+            for sentence in text:
+                # TODO: if we change the predict_tagset mechanism, will have to change this
+                sentence_gold = [x[predict_tagset] for x in batch.tags[tag_idx]]
+                sentence_pred = preds[tag_idx]
+                tag_idx += 1
+                for word, gold, pred in zip(sentence, sentence_gold, sentence_pred):
+                    fout.write("%s\t%s\t%s\n" % (word, gold, pred))
+                fout.write("\n")
+
+def evaluate(args):
+    # file paths
+    model_file = model_file_name(args)
+
+    loaded_args, trainer, vocab = load_model(args, model_file)
+    return evaluate_model(loaded_args, trainer, vocab, args['eval_file'])
+
+def evaluate_model(loaded_args, trainer, vocab, eval_file):
+    if loaded_args['log_norms']:
+        trainer.model.log_norms()
+
+    model_file = os.path.join(loaded_args['save_dir'], loaded_args['save_name'])
+    logger.debug("Loaded model for eval from %s", model_file)
+    logger.debug("Using the %d tagset for evaluation", loaded_args['predict_tagset'])
+
+    # load data
+    logger.info("Loading data with batch size {}...".format(loaded_args['batch_size']))
+    with open(eval_file) as fin:
+        doc = Document(json.load(fin))
+    batch = DataLoader(doc, loaded_args['batch_size'], loaded_args, vocab=vocab, evaluation=True, bert_tokenizer=trainer.model.bert_tokenizer)
+    bioes_to_bio = loaded_args['train_scheme'] == 'bio' and loaded_args['scheme'] == 'bioes'
+    warn_missing_tags(trainer.vocab['tag'], batch.tags, "eval_file", bioes_to_bio=bioes_to_bio)
+
+    logger.info("Start evaluation...")
+    preds = []
+    for i, b in enumerate(batch):
+        preds += trainer.predict(b)
+
+    gold_tags = batch.tags
+    # TODO: might still want to add multiple layers of tag evaluation to the scorer
+    gold_tags = [[x[trainer.args['predict_tagset']] for x in tags] for tags in gold_tags]
+
+    _, _, score, entity_f1 = scorer.score_by_entity(preds, gold_tags, ignore_tags=loaded_args['ignore_tag_scores'])
+    _, _, _, confusion = scorer.score_by_token(preds, gold_tags, ignore_tags=loaded_args['ignore_tag_scores'])
+    logger.info("Weighted f1 for non-O tokens: %5f", confusion_to_weighted_f1(confusion, exclude=["O"]))
+
+    logger.info("NER tagger score: %s %s %s %.2f", loaded_args['shorthand'], model_file, eval_file, score*100)
+    entity_f1_lines = ["%s: %.2f" % (x, y*100) for x, y in entity_f1.items()]
+    logger.info("NER Entity F1 scores:\n  %s", "\n  ".join(entity_f1_lines))
+    logger.info("NER token confusion matrix:\n{}".format(format_confusion(confusion)))
+
+    if loaded_args['eval_output_file']:
+        write_ner_results(loaded_args['eval_output_file'], batch, preds, trainer.args['predict_tagset'])
+
+    return confusion
+
+def load_model(args, model_file):
+    # load model
+    charlm_args = {}
+    if 'charlm_forward_file' in args:
+        charlm_args['charlm_forward_file'] = args['charlm_forward_file']
+    if 'charlm_backward_file' in args:
+        charlm_args['charlm_backward_file'] = args['charlm_backward_file']
+    if args['predict_tagset'] is not None:
+        charlm_args['predict_tagset'] = args['predict_tagset']
+    pretrain = load_pretrain(args)
+    trainer = Trainer(args=charlm_args, model_file=model_file, pretrain=pretrain, device=args['device'], train_classifier_only=args['train_classifier_only'])
+    loaded_args, vocab = trainer.args, trainer.vocab
+
+    # load config
+    for k in args:
+        if k.endswith('_dir') or k.endswith('_file') or k in ['batch_size', 'ignore_tag_scores', 'log_norms', 'mode', 'scheme', 'shorthand']:
+            loaded_args[k] = args[k]
+    save_dir, save_name = os.path.split(model_file)
+    args['save_dir'] = save_dir
+    args['save_name'] = save_name
+    return loaded_args, trainer, vocab
+
+
+if __name__ == '__main__':
+    main()

stanza/stanza/models/tagger.py

@@ -0,0 +1,461 @@
+"""
+Entry point for training and evaluating a POS/morphological features tagger.
+
+This tagger uses highway BiLSTM layers with character and word-level representations, and biaffine classifiers
+to produce consistent POS and UFeats predictions.
+For details please refer to paper: https://nlp.stanford.edu/pubs/qi2018universal.pdf.
+"""
+
+import argparse
+import logging
+import os
+import time
+import zipfile
+
+import numpy as np
+import torch
+from torch import nn, optim
+
+from stanza.models.pos.data import Dataset, ShuffledDataset
+from stanza.models.pos.trainer import Trainer
+from stanza.models.pos import scorer
+from stanza.models.common import utils
+from stanza.models.common import pretrain
+from stanza.models.common.doc import *
+from stanza.models.common.foundation_cache import FoundationCache
+from stanza.models.common.peft_config import add_peft_args, resolve_peft_args
+from stanza.models import _training_logging
+from stanza.utils.conll import CoNLL
+
+logger = logging.getLogger('stanza')
+
+def build_argparse():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--data_dir', type=str, default='data/pos', help='Root dir for saving models.')
+    parser.add_argument('--wordvec_dir', type=str, default='extern_data/wordvec', help='Directory of word vectors.')
+    parser.add_argument('--wordvec_file', type=str, default=None, help='Word vectors filename.')
+    parser.add_argument('--wordvec_pretrain_file', type=str, default=None, help='Exact name of the pretrain file to read')
+    parser.add_argument('--train_file', type=str, default=None, help='Input file for training.')
+    parser.add_argument('--eval_file', type=str, default=None, help='Input file for scoring.')
+    parser.add_argument('--output_file', type=str, default=None, help='Output CoNLL-U file.')
+    parser.add_argument('--no_gold_labels', dest='gold_labels', action='store_false', help="Don't score the eval file - perhaps it has no gold labels, for example.  Cannot be used at training time")
+
+    parser.add_argument('--mode', default='train', choices=['train', 'predict'])
+    parser.add_argument('--lang', type=str, help='Language')
+    parser.add_argument('--shorthand', type=str, help="Treebank shorthand")
+
+    parser.add_argument('--hidden_dim', type=int, default=200)
+    parser.add_argument('--char_hidden_dim', type=int, default=400)
+    parser.add_argument('--deep_biaff_hidden_dim', type=int, default=400)
+    parser.add_argument('--composite_deep_biaff_hidden_dim', type=int, default=100)
+    parser.add_argument('--word_emb_dim', type=int, default=75, help='Dimension of the finetuned word embedding.  Set to 0 to turn off')
+    parser.add_argument('--word_cutoff', type=int, default=7, help='How common a word must be to include it in the finetuned word embedding')
+    parser.add_argument('--char_emb_dim', type=int, default=100)
+    parser.add_argument('--tag_emb_dim', type=int, default=50)
+    parser.add_argument('--charlm_transform_dim', type=int, default=None, help='Transform the pretrained charlm to this dimension.  If not set, no transform is used')
+    parser.add_argument('--transformed_dim', type=int, default=125)
+    parser.add_argument('--num_layers', type=int, default=2)
+    parser.add_argument('--char_num_layers', type=int, default=1)
+    parser.add_argument('--pretrain_max_vocab', type=int, default=250000)
+    parser.add_argument('--word_dropout', type=float, default=0.33)
+    parser.add_argument('--dropout', type=float, default=0.5)
+    parser.add_argument('--rec_dropout', type=float, default=0, help="Recurrent dropout")
+    parser.add_argument('--char_rec_dropout', type=float, default=0, help="Recurrent dropout")
+
+    # TODO: refactor charlm arguments for models which use it?
+    parser.add_argument('--no_char', dest='char', action='store_false', help="Turn off character model.")
+    parser.add_argument('--char_bidirectional', dest='char_bidirectional', action='store_true', help="Use a bidirectional version of the non-pretrained charlm.  Doesn't help much, makes the models larger")
+    parser.add_argument('--char_lowercase', dest='char_lowercase', action='store_true', help="Use lowercased characters in character model.")
+    parser.add_argument('--charlm', action='store_true', help="Turn on contextualized char embedding using pretrained character-level language model.")
+    parser.add_argument('--charlm_save_dir', type=str, default='saved_models/charlm', help="Root dir for pretrained character-level language model.")
+    parser.add_argument('--charlm_shorthand', type=str, default=None, help="Shorthand for character-level language model training corpus.")
+    parser.add_argument('--charlm_forward_file', type=str, default=None, help="Exact path to use for forward charlm")
+    parser.add_argument('--charlm_backward_file', type=str, default=None, help="Exact path to use for backward charlm")
+
+    parser.add_argument('--bert_model', type=str, default=None, help="Use an external bert model (requires the transformers package)")
+    parser.add_argument('--no_bert_model', dest='bert_model', action="store_const", const=None, help="Don't use bert")
+    parser.add_argument('--bert_hidden_layers', type=int, default=None, help="How many layers of hidden state to use from the transformer")
+    parser.add_argument('--bert_finetune', default=False, action='store_true', help='Finetune the bert (or other transformer)')
+    parser.add_argument('--no_bert_finetune', dest='bert_finetune', action='store_false', help="Don't finetune the bert (or other transformer)")
+    parser.add_argument('--bert_learning_rate', default=1.0, type=float, help='Scale the learning rate for transformer finetuning by this much')
+
+    parser.add_argument('--no_pretrain', dest='pretrain', action='store_false', help="Turn off pretrained embeddings.")
+    parser.add_argument('--share_hid', action='store_true', help="Share hidden representations for UPOS, XPOS and UFeats.")
+    parser.set_defaults(share_hid=False)
+
+    parser.add_argument('--sample_train', type=float, default=1.0, help='Subsample training data.')
+    parser.add_argument('--optim', type=str, default='adam', help='sgd, adagrad, adam, adamw, adamax, or adadelta.  madgrad as an optional dependency')
+    parser.add_argument('--second_optim', type=str, default='amsgrad', help='Optimizer for the second half of training.  Default is Adam with AMSGrad')
+    parser.add_argument('--second_optim_reload', default=False, action='store_true', help='Reload the best model instead of continuing from current model if the first optimizer stalls out.  This does not seem to help, but might be useful for further experiments')
+    parser.add_argument('--no_second_optim', action='store_const', const=None, dest='second_optim', help="Don't use a second optimizer - only use the first optimizer")
+    parser.add_argument('--lr', type=float, default=3e-3, help='Learning rate')
+    parser.add_argument('--second_lr', type=float, default=None, help='Alternate learning rate for the second optimizer')
+    parser.add_argument('--initial_weight_decay', type=float, default=None, help='Optimizer weight decay for the first optimizer')
+    parser.add_argument('--second_weight_decay', type=float, default=None, help='Optimizer weight decay for the second optimizer')
+    parser.add_argument('--beta2', type=float, default=0.95)
+
+    parser.add_argument('--max_steps', type=int, default=50000)
+    parser.add_argument('--eval_interval', type=int, default=100)
+    parser.add_argument('--fix_eval_interval', dest='adapt_eval_interval', action='store_false', \
+            help="Use fixed evaluation interval for all treebanks, otherwise by default the interval will be increased for larger treebanks.")
+    parser.add_argument('--max_steps_before_stop', type=int, default=3000, help='Changes learning method or early terminates after this many steps if the dev scores are not improving')
+    parser.add_argument('--batch_size', type=int, default=250)
+    parser.add_argument('--batch_maximum_tokens', type=int, default=5000, help='When run in a Pipeline, limit a batch to this many tokens to help avoid OOM for long sentences')
+    parser.add_argument('--max_grad_norm', type=float, default=1.0, help='Gradient clipping.')
+    parser.add_argument('--log_step', type=int, default=20, help='Print log every k steps.')
+    parser.add_argument('--log_norms', action='store_true', default=False, help='Log the norms of all the parameters (noisy!)')
+    parser.add_argument('--save_dir', type=str, default='saved_models/pos', help='Root dir for saving models.')
+    parser.add_argument('--save_name', type=str, default="{shorthand}_{embedding}_tagger.pt", help="File name to save the model")
+    parser.add_argument('--save_each', default=False, action='store_true', help="Save each checkpoint to its own model.  Will take up a bunch of space")
+
+    parser.add_argument('--seed', type=int, default=1234)
+    add_peft_args(parser)
+    utils.add_device_args(parser)
+
+    parser.add_argument('--augment_nopunct', type=float, default=None, help='Augment the training data by copying this fraction of punct-ending sentences as non-punct.  Default of None will aim for roughly 50%%')
+
+    parser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name')
+    parser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+    return parser
+
+def parse_args(args=None):
+    parser = build_argparse()
+    args = parser.parse_args(args=args)
+    resolve_peft_args(args, logger)
+
+    if args.augment_nopunct is None:
+        args.augment_nopunct = 0.25
+
+    if args.wandb_name:
+        args.wandb = True
+
+    args = vars(args)
+    return args
+
+def main(args=None):
+    args = parse_args(args=args)
+
+    utils.set_random_seed(args['seed'])
+
+    logger.info("Running tagger in {} mode".format(args['mode']))
+
+    if args['mode'] == 'train':
+        train(args)
+    else:
+        evaluate(args)
+
+def model_file_name(args):
+    return utils.standard_model_file_name(args, "tagger")
+
+def save_each_file_name(args):
+    model_file = model_file_name(args)
+    pieces = os.path.splitext(model_file)
+    return pieces[0] + "_%05d" + pieces[1]
+
+def load_pretrain(args):
+    pt = None
+    if args['pretrain']:
+        pretrain_file = pretrain.find_pretrain_file(args['wordvec_pretrain_file'], args['save_dir'], args['shorthand'], args['lang'])
+        if os.path.exists(pretrain_file):
+            vec_file = None
+        else:
+            vec_file = args['wordvec_file'] if args['wordvec_file'] else utils.get_wordvec_file(args['wordvec_dir'], args['shorthand'])
+        pt = pretrain.Pretrain(pretrain_file, vec_file, args['pretrain_max_vocab'])
+    return pt
+
+def get_eval_type(dev_batch):
+    """
+    If there is only one column to score in the dev set, use that instead of AllTags
+    """
+    if dev_batch.has_xpos and not dev_batch.has_upos and not dev_batch.has_feats:
+        return "XPOS"
+    elif dev_batch.has_upos and not dev_batch.has_xpos and not dev_batch.has_feats:
+        return "UPOS"
+    else:
+        return "AllTags"
+
+def load_training_data(args, pretrain):
+    train_docs = []
+    raw_train_files = args['train_file'].split(";")
+    train_files = []
+    for train_file in raw_train_files:
+        if zipfile.is_zipfile(train_file):
+            logger.info("Decompressing %s" % train_file)
+            with zipfile.ZipFile(train_file) as zin:
+                for zipped_train_file in zin.namelist():
+                    with zin.open(zipped_train_file) as fin:
+                        logger.info("Reading %s from %s" % (zipped_train_file, train_file))
+                        train_str = fin.read()
+                        train_str = train_str.decode("utf-8")
+                        train_file_data, _, _ = CoNLL.conll2dict(input_str=train_str)
+                        logger.info("Train File {} from {}, Data Size: {}".format(zipped_train_file, train_file, len(train_file_data)))
+                        train_docs.append(Document(train_file_data))
+                        train_files.append("%s %s" % (train_file, zipped_train_file))
+        else:
+            logger.info("Reading %s" % train_file)
+            # train_data is now a list of sentences, where each sentence is a
+            # list of words, in which each word is a dict of conll attributes
+            train_file_data, _, _ = CoNLL.conll2dict(input_file=train_file)
+            logger.info("Train File {}, Data Size: {}".format(train_file, len(train_file_data)))
+            train_docs.append(Document(train_file_data))
+            train_files.append(train_file)
+    if sum(len(x.sentences) for x in train_docs) == 0:
+        raise RuntimeError("Training data for the tagger is empty: %s" % args['train_file'])
+    # we want to ensure that the model is able te output _ for empty columns,
+    # but create batches whereby if a doc has upos/xpos tags we include them all.
+    # therefore, we create seperate datasets and loaders for each input training file,
+    # which will ensure the system be able to see batches with both upos available
+    # and upos unavailable depending on what the availability in the file is.
+    vocab = Dataset.init_vocab(train_docs, args)
+    train_data = [Dataset(i, args, pretrain, vocab=vocab, evaluation=False)
+                  for i in train_docs]
+    for train_file, td in zip(train_files, train_data):
+        if not td.has_upos:
+            logger.info("No UPOS in %s" % train_file)
+        if not td.has_xpos:
+            logger.info("No XPOS in %s" % train_file)
+        if not td.has_feats:
+            logger.info("No feats in %s" % train_file)
+
+    # reject partially tagged upos or xpos documents
+    # otherwise, the model will learn to output blanks for some words,
+    # which is probably a confusing result
+    # (and definitely throws off the depparse)
+    # another option would be to treat those as masked out
+    for td_idx, td in enumerate(train_data):
+        if td.has_upos:
+            upos_data = td.doc.get(UPOS, as_sentences=True)
+            for sentence_idx, sentence in enumerate(upos_data):
+                for word_idx, upos in enumerate(sentence):
+                    if upos == '_' or upos is None:
+                        conll = "{:C}".format(td.doc.sentences[sentence_idx])
+                        raise RuntimeError("Found a blank tag in the UPOS at sentence %d word %d of %s.\n%s" % ((sentence_idx+1), (word_idx+1), train_files[td_idx], conll))
+
+    # here we make sure the model will learn to output _ for empty columns
+    # if *any* dataset has data for the upos, xpos, or feature column,
+    # we consider that data enough to train the model on that column
+    # otherwise, we want to train the model to always output blanks
+    if not any(td.has_upos for td in train_data):
+        for td in train_data:
+            td.has_upos = True
+    if not any(td.has_xpos for td in train_data):
+        for td in train_data:
+            td.has_xpos = True
+    if not any(td.has_feats for td in train_data):
+        for td in train_data:
+            td.has_feats = True
+    # calculate the batches
+    train_batches = ShuffledDataset(train_data, args["batch_size"])
+    return vocab, train_data, train_batches
+
+def train(args):
+    model_file = model_file_name(args)
+    utils.ensure_dir(os.path.split(model_file)[0])
+
+    if args['save_each']:
+        # so models.pt -> models_0001.pt, etc
+        model_save_each_file = save_each_file_name(args)
+        logger.info("Saving each checkpoint to %s" % model_save_each_file)
+
+    # load pretrained vectors if needed
+    pretrain = load_pretrain(args)
+
+    if args['charlm']:
+        if args['charlm_shorthand'] is None:
+            raise ValueError("CharLM Shorthand is required for loading pretrained CharLM model...")
+        logger.info('Using pretrained contextualized char embedding')
+        if not args['charlm_forward_file']:
+            args['charlm_forward_file'] = '{}/{}_forward_charlm.pt'.format(args['charlm_save_dir'], args['charlm_shorthand'])
+        if not args['charlm_backward_file']:
+            args['charlm_backward_file'] = '{}/{}_backward_charlm.pt'.format(args['charlm_save_dir'], args['charlm_shorthand'])
+
+    # load data
+    logger.info("Loading data with batch size {}...".format(args['batch_size']))
+    vocab, train_data, train_batches = load_training_data(args, pretrain)
+
+    dev_doc = CoNLL.conll2doc(input_file=args['eval_file'])
+    dev_data = Dataset(dev_doc, args, pretrain, vocab=vocab, evaluation=True, sort_during_eval=True)
+    dev_batch = dev_data.to_loader(batch_size=args["batch_size"])
+
+    eval_type = get_eval_type(dev_data)
+
+    # pred and gold path
+    system_pred_file = args['output_file']
+
+    # skip training if the language does not have training or dev data
+    # sum(...) to check if all of the training files are empty
+    if sum(len(td) for td in train_data) == 0 or len(dev_data) == 0:
+        logger.info("Skip training because no data available...")
+        return
+
+    if args['wandb']:
+        import wandb
+        wandb_name = args['wandb_name'] if args['wandb_name'] else "%s_tagger" % args['shorthand']
+        wandb.init(name=wandb_name, config=args)
+        wandb.run.define_metric('train_loss', summary='min')
+        wandb.run.define_metric('dev_score', summary='max')
+
+    logger.info("Training tagger...")
+    foundation_cache = FoundationCache()
+    trainer = Trainer(args=args, vocab=vocab, pretrain=pretrain, device=args['device'], foundation_cache=foundation_cache)
+
+    global_step = 0
+    max_steps = args['max_steps']
+    dev_score_history = []
+    best_dev_preds = []
+    current_lr = args['lr']
+    global_start_time = time.time()
+    format_str = 'Finished STEP {}/{}, loss = {:.6f} ({:.3f} sec/batch), lr: {:.6f}'
+
+    if args['adapt_eval_interval']:
+        args['eval_interval'] = utils.get_adaptive_eval_interval(dev_data.num_examples, 2000, args['eval_interval'])
+        logger.info("Evaluating the model every {} steps...".format(args['eval_interval']))
+
+    if args['save_each']:
+        logger.info("Saving initial checkpoint to %s" % (model_save_each_file % global_step))
+        trainer.save(model_save_each_file % global_step)
+
+    using_amsgrad = False
+    last_best_step = 0
+    # start training
+    train_loss = 0
+    if args['log_norms']:
+        trainer.model.log_norms()
+    while True:
+        do_break = False
+        for i, batch in enumerate(train_batches):
+            start_time = time.time()
+            global_step += 1
+            loss = trainer.update(batch, eval=False) # update step
+            train_loss += loss
+            if global_step % args['log_step'] == 0:
+                duration = time.time() - start_time
+                logger.info(format_str.format(global_step, max_steps, loss, duration, current_lr))
+                if args['log_norms']:
+                    trainer.model.log_norms()
+
+            if global_step % args['eval_interval'] == 0:
+                # eval on dev
+                logger.info("Evaluating on dev set...")
+                dev_preds = []
+                indices = []
+                for batch in dev_batch:
+                    preds = trainer.predict(batch)
+                    dev_preds += preds
+                    indices.extend(batch[-1])
+                dev_preds = utils.unsort(dev_preds, indices)
+                dev_data.doc.set([UPOS, XPOS, FEATS], [y for x in dev_preds for y in x])
+                CoNLL.write_doc2conll(dev_data.doc, system_pred_file)
+
+                _, _, dev_score = scorer.score(system_pred_file, args['eval_file'], eval_type=eval_type)
+
+                train_loss = train_loss / args['eval_interval'] # avg loss per batch
+                logger.info("step {}: train_loss = {:.6f}, dev_score = {:.4f}".format(global_step, train_loss, dev_score))
+
+                if args['wandb']:
+                    wandb.log({'train_loss': train_loss, 'dev_score': dev_score})
+
+                train_loss = 0
+
+                if args['save_each']:
+                    logger.info("Saving checkpoint to %s" % (model_save_each_file % global_step))
+                    trainer.save(model_save_each_file % global_step)
+
+                # save best model
+                if len(dev_score_history) == 0 or dev_score > max(dev_score_history):
+                    last_best_step = global_step
+                    trainer.save(model_file)
+                    logger.info("new best model saved.")
+                    best_dev_preds = dev_preds
+
+                dev_score_history += [dev_score]
+
+            if global_step - last_best_step >= args['max_steps_before_stop']:
+                if not using_amsgrad and args['second_optim'] is not None:
+                    logger.info("Switching to second optimizer: {}".format(args['second_optim']))
+                    if args['second_optim_reload']:
+                        logger.info('Reloading best model to continue from current local optimum')
+                        trainer = Trainer(args=args, vocab=trainer.vocab, pretrain=pretrain, model_file=model_file, device=args['device'], foundation_cache=foundation_cache)
+                    last_best_step = global_step
+                    using_amsgrad = True
+                    lr = args['second_lr']
+                    if lr is None:
+                        lr = args['lr']
+                    trainer.optimizer = utils.get_optimizer(args['second_optim'], trainer.model, lr=lr, betas=(.9, args['beta2']), eps=1e-6, weight_decay=args['second_weight_decay'])
+                else:
+                    logger.info("Early termination: have not improved in {} steps".format(args['max_steps_before_stop']))
+                    do_break = True
+                    break
+
+            if global_step >= args['max_steps']:
+                do_break = True
+                break
+
+        if do_break: break
+
+    logger.info("Training ended with {} steps.".format(global_step))
+
+    if args['wandb']:
+        wandb.finish()
+
+    if len(dev_score_history) > 0:
+        best_f, best_eval = max(dev_score_history)*100, np.argmax(dev_score_history)+1
+        logger.info("Best dev F1 = {:.2f}, at iteration = {}".format(best_f, best_eval * args['eval_interval']))
+    else:
+        logger.info("Dev set never evaluated.  Saving final model.")
+        trainer.save(model_file)
+
+
+def evaluate(args):
+    # file paths
+    model_file = model_file_name(args)
+
+    pretrain = load_pretrain(args)
+
+    load_args = {'charlm_forward_file': args.get('charlm_forward_file', None),
+                 'charlm_backward_file': args.get('charlm_backward_file', None)}
+
+    # load model
+    logger.info("Loading model from: {}".format(model_file))
+    trainer = Trainer(pretrain=pretrain, model_file=model_file, device=args['device'], args=load_args)
+    evaluate_trainer(args, trainer, pretrain)
+
+def evaluate_trainer(args, trainer, pretrain):
+    system_pred_file = args['output_file']
+    loaded_args, vocab = trainer.args, trainer.vocab
+
+    # load config
+    for k in args:
+        if k.endswith('_dir') or k.endswith('_file') or k in ['shorthand'] or k == 'mode':
+            loaded_args[k] = args[k]
+
+    # load data
+    logger.info("Loading data with batch size {}...".format(args['batch_size']))
+    doc = CoNLL.conll2doc(input_file=args['eval_file'])
+    dev_data = Dataset(doc, loaded_args, pretrain, vocab=vocab, evaluation=True, sort_during_eval=True)
+    dev_batch = dev_data.to_loader(batch_size=args['batch_size'])
+    eval_type = get_eval_type(dev_data)
+    if len(dev_batch) > 0:
+        logger.info("Start evaluation...")
+        preds = []
+        indices = []
+        with torch.no_grad():
+            for b in dev_batch:
+                preds += trainer.predict(b)
+                indices.extend(b[-1])
+    else:
+        # skip eval if dev data does not exist
+        preds = []
+    preds = utils.unsort(preds, indices)
+
+    # write to file and score
+    dev_data.doc.set([UPOS, XPOS, FEATS], [y for x in preds for y in x])
+    CoNLL.write_doc2conll(dev_data.doc, system_pred_file)
+
+    if args['gold_labels']:
+        _, _, score = scorer.score(system_pred_file, args['eval_file'], eval_type=eval_type)
+
+        logger.info("POS Tagger score: %s %.2f", args['shorthand'], score*100)
+
+if __name__ == '__main__':
+    main()

stanza/stanza/models/tokenizer.py

@@ -0,0 +1,258 @@
+"""
+Entry point for training and evaluating a neural tokenizer.
+
+This tokenizer treats tokenization and sentence segmentation as a tagging problem, and uses a combination of
+recurrent and convolutional architectures.
+For details please refer to paper: https://nlp.stanford.edu/pubs/qi2018universal.pdf.
+
+Updated: This new version of tokenizer model incorporates the dictionary feature, especially useful for languages that
+have multi-syllable words such as Vietnamese, Chinese or Thai. In summary, a lexicon contains all unique words found in 
+training dataset and external lexicon (if any) is created during training and saved alongside the model after training.
+Using this lexicon, a dictionary is created which includes "words", "prefixes" and "suffixes" sets. During data preparation,
+dictionary features are extracted at each character position, to "look ahead" and "look backward" to see if any words formed
+found in the dictionary. The window size (or the dictionary feature length) is defined at the 95-percentile among all the existing
+words in the lexicon, this is to eliminate the less frequent but long words (avoid having a high-dimension feat vector). Prefixes 
+and suffixes are used to stop early during the window-dictionary checking process.  
+"""
+
+import argparse
+from copy import copy
+import logging
+import random
+import numpy as np
+import os
+import torch
+import json
+from stanza.models.common import utils
+from stanza.models.tokenization.trainer import Trainer
+from stanza.models.tokenization.data import DataLoader, TokenizationDataset
+from stanza.models.tokenization.utils import load_mwt_dict, eval_model, output_predictions, load_lexicon, create_dictionary
+from stanza.models import _training_logging
+
+logger = logging.getLogger('stanza')
+
+def build_argparse():
+    """
+    If args == None, the system args are used.
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--txt_file', type=str, help="Input plaintext file")
+    parser.add_argument('--label_file', type=str, default=None, help="Character-level label file")
+    parser.add_argument('--mwt_json_file', type=str, default=None, help="JSON file for MWT expansions")
+    parser.add_argument('--conll_file', type=str, default=None, help="CoNLL file for output")
+    parser.add_argument('--dev_txt_file', type=str, help="(Train only) Input plaintext file for the dev set")
+    parser.add_argument('--dev_label_file', type=str, default=None, help="(Train only) Character-level label file for the dev set")
+    parser.add_argument('--dev_conll_gold', type=str, default=None, help="(Train only) CoNLL-U file for the dev set for early stopping")
+    parser.add_argument('--lang', type=str, help="Language")
+    parser.add_argument('--shorthand', type=str, help="UD treebank shorthand")
+
+    parser.add_argument('--mode', default='train', choices=['train', 'predict'])
+    parser.add_argument('--skip_newline', action='store_true', help="Whether to skip newline characters in input. Particularly useful for languages like Chinese.")
+
+    parser.add_argument('--emb_dim', type=int, default=32, help="Dimension of unit embeddings")
+    parser.add_argument('--hidden_dim', type=int, default=64, help="Dimension of hidden units")
+    parser.add_argument('--conv_filters', type=str, default="1,9", help="Configuration of conv filters. ,, separates layers and , separates filter sizes in the same layer.")
+    parser.add_argument('--no-residual', dest='residual', action='store_false', help="Add linear residual connections")
+    parser.add_argument('--no-hierarchical', dest='hierarchical', action='store_false', help="\"Hierarchical\" RNN tokenizer")
+    parser.add_argument('--hier_invtemp', type=float, default=0.5, help="Inverse temperature used in propagating tokenization predictions between RNN layers")
+    parser.add_argument('--input_dropout', action='store_true', help="Dropout input embeddings as well")
+    parser.add_argument('--conv_res', type=str, default=None, help="Convolutional residual layers for the RNN")
+    parser.add_argument('--rnn_layers', type=int, default=1, help="Layers of RNN in the tokenizer")
+    parser.add_argument('--use_dictionary', action='store_true', help="Use dictionary feature. The lexicon is created using the training data and external dict (if any) expected to be found under the same folder of training dataset, formatted as SHORTHAND-externaldict.txt where each line in this file is a word. For example, data/tokenize/zh_gsdsimp-externaldict.txt")
+
+    parser.add_argument('--max_grad_norm', type=float, default=1.0, help="Maximum gradient norm to clip to")
+    parser.add_argument('--anneal', type=float, default=.999, help="Anneal the learning rate by this amount when dev performance deteriorate")
+    parser.add_argument('--anneal_after', type=int, default=2000, help="Anneal the learning rate no earlier than this step")
+    parser.add_argument('--lr0', type=float, default=2e-3, help="Initial learning rate")
+    parser.add_argument('--dropout', type=float, default=0.33, help="Dropout probability")
+    parser.add_argument('--unit_dropout', type=float, default=0.33, help="Unit dropout probability")
+    parser.add_argument('--feat_dropout', type=float, default=0.05, help="Features dropout probability for each element in feature vector")
+    parser.add_argument('--feat_unit_dropout', type=float, default=0.33, help="The whole feature of units dropout probability")
+    parser.add_argument('--tok_noise', type=float, default=0.02, help="Probability to induce noise to the input of the higher RNN")
+    parser.add_argument('--sent_drop_prob', type=float, default=0.2, help="Probability to drop sentences at the end of batches during training uniformly at random.  Idea is to fake paragraph endings.")
+    parser.add_argument('--last_char_drop_prob', type=float, default=0.2, help="Probability to drop the last char of a block of text during training, uniformly at random.  Idea is to fake a document ending w/o sentence final punctuation, hopefully to avoid the tokenizer learning to always tokenize the last character as a period")
+    parser.add_argument('--weight_decay', type=float, default=0.0, help="Weight decay")
+    parser.add_argument('--max_seqlen', type=int, default=100, help="Maximum sequence length to consider at a time")
+    parser.add_argument('--batch_size', type=int, default=32, help="Batch size to use")
+    parser.add_argument('--epochs', type=int, default=10, help="Total epochs to train the model for")
+    parser.add_argument('--steps', type=int, default=50000, help="Steps to train the model for, if unspecified use epochs")
+    parser.add_argument('--report_steps', type=int, default=20, help="Update step interval to report loss")
+    parser.add_argument('--shuffle_steps', type=int, default=100, help="Step interval to shuffle each paragraph in the generator")
+    parser.add_argument('--eval_steps', type=int, default=200, help="Step interval to evaluate the model on the dev set for early stopping")
+    parser.add_argument('--max_steps_before_stop', type=int, default=5000, help='Early terminates after this many steps if the dev scores are not improving')
+    parser.add_argument('--save_name', type=str, default=None, help="File name to save the model")
+    parser.add_argument('--load_name', type=str, default=None, help="File name to load a saved model")
+    parser.add_argument('--save_dir', type=str, default='saved_models/tokenize', help="Directory to save models in")
+    utils.add_device_args(parser)
+    parser.add_argument('--seed', type=int, default=1234)
+
+    parser.add_argument('--use_mwt', dest='use_mwt', default=None, action='store_true', help='Whether or not to include mwt output layers.  If set to None, this will be determined by examining the training data for MWTs')
+    parser.add_argument('--no_use_mwt', dest='use_mwt', action='store_false', help='Whether or not to include mwt output layers')
+
+    parser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name')
+    parser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+    return parser
+
+def parse_args(args=None):
+    parser = build_argparse()
+    args = parser.parse_args(args=args)
+
+    if args.wandb_name:
+        args.wandb = True
+
+    args = vars(args)
+    return args
+
+def model_file_name(args):
+    if args['save_name'] is not None:
+        save_name = args['save_name']
+    else:
+        save_name = args['shorthand'] + "_tokenizer.pt"
+
+    if not os.path.exists(os.path.join(args['save_dir'], save_name)) and os.path.exists(save_name):
+        return save_name
+    return os.path.join(args['save_dir'], save_name)
+
+def main(args=None):
+    args = parse_args(args=args)
+
+    utils.set_random_seed(args['seed'])
+
+    logger.info("Running tokenizer in {} mode".format(args['mode']))
+
+    args['feat_funcs'] = ['space_before', 'capitalized', 'numeric', 'end_of_para', 'start_of_para']
+    args['feat_dim'] = len(args['feat_funcs'])
+    args['save_name'] = model_file_name(args)
+    utils.ensure_dir(os.path.split(args['save_name'])[0])
+
+    if args['mode'] == 'train':
+        train(args)
+    else:
+        evaluate(args)
+
+def train(args):
+    if args['use_dictionary']:
+        #load lexicon
+        lexicon, args['num_dict_feat'] = load_lexicon(args)
+        #create the dictionary
+        dictionary = create_dictionary(lexicon)
+        #adjust the feat_dim
+        args['feat_dim'] += args['num_dict_feat']*2
+    else:
+        args['num_dict_feat'] = 0
+        lexicon=None
+        dictionary=None
+
+    mwt_dict = load_mwt_dict(args['mwt_json_file'])
+
+    train_input_files = {
+            'txt': args['txt_file'],
+            'label': args['label_file']
+            }
+    train_batches = DataLoader(args, input_files=train_input_files, dictionary=dictionary)
+    vocab = train_batches.vocab
+
+    args['vocab_size'] = len(vocab)
+
+    dev_input_files = {
+            'txt': args['dev_txt_file'],
+            'label': args['dev_label_file']
+            }
+    dev_batches = TokenizationDataset(args, input_files=dev_input_files, vocab=vocab, evaluation=True, dictionary=dictionary)
+
+    if args['use_mwt'] is None:
+        args['use_mwt'] = train_batches.has_mwt()
+        logger.info("Found {}mwts in the training data.  Setting use_mwt to {}".format(("" if args['use_mwt'] else "no "), args['use_mwt']))
+
+    trainer = Trainer(args=args, vocab=vocab, lexicon=lexicon, dictionary=dictionary, device=args['device'])
+
+    if args['load_name'] is not None:
+        load_name = os.path.join(args['save_dir'], args['load_name'])
+        trainer.load(load_name)
+    trainer.change_lr(args['lr0'])
+
+    N = len(train_batches)
+    steps = args['steps'] if args['steps'] is not None else int(N * args['epochs'] / args['batch_size'] + .5)
+    lr = args['lr0']
+
+    prev_dev_score = -1
+    best_dev_score = -1
+    best_dev_step = -1
+
+    if args['wandb']:
+        import wandb
+        wandb_name = args['wandb_name'] if args['wandb_name'] else "%s_tokenizer" % args['shorthand']
+        wandb.init(name=wandb_name, config=args)
+        wandb.run.define_metric('train_loss', summary='min')
+        wandb.run.define_metric('dev_score', summary='max')
+
+
+    for step in range(1, steps+1):
+        batch = train_batches.next(unit_dropout=args['unit_dropout'], feat_unit_dropout = args['feat_unit_dropout'])
+
+        loss = trainer.update(batch)
+        if step % args['report_steps'] == 0:
+            logger.info("Step {:6d}/{:6d} Loss: {:.3f}".format(step, steps, loss))
+            if args['wandb']:
+                wandb.log({'train_loss': loss}, step=step)
+
+        if args['shuffle_steps'] > 0 and step % args['shuffle_steps'] == 0:
+            train_batches.shuffle()
+
+        if step % args['eval_steps'] == 0:
+            dev_score = eval_model(args, trainer, dev_batches, vocab, mwt_dict)
+            if args['wandb']:
+                wandb.log({'dev_score': dev_score}, step=step)
+            reports = ['Dev score: {:6.3f}'.format(dev_score * 100)]
+            if step >= args['anneal_after'] and dev_score < prev_dev_score:
+                reports += ['lr: {:.6f} -> {:.6f}'.format(lr, lr * args['anneal'])]
+                lr *= args['anneal']
+                trainer.change_lr(lr)
+
+            prev_dev_score = dev_score
+
+            if dev_score > best_dev_score:
+                reports += ['New best dev score!']
+                best_dev_score = dev_score
+                best_dev_step = step
+                trainer.save(args['save_name'])
+            elif best_dev_step > 0 and step - best_dev_step > args['max_steps_before_stop']:
+                reports += ['Stopping training after {} steps with no improvement'.format(step - best_dev_step)]
+                logger.info('\t'.join(reports))
+                break
+
+            logger.info('\t'.join(reports))
+
+    if args['wandb']:
+        wandb.finish()
+
+    if best_dev_step > -1:
+        logger.info('Best dev score={} at step {}'.format(best_dev_score, best_dev_step))
+    else:
+        logger.info('Dev set never evaluated.  Saving final model')
+        trainer.save(args['save_name'])
+
+def evaluate(args):
+    mwt_dict = load_mwt_dict(args['mwt_json_file'])
+    trainer = Trainer(model_file=args['load_name'] or args['save_name'], device=args['device'])
+    loaded_args, vocab = trainer.args, trainer.vocab
+
+    for k in loaded_args:
+        if not k.endswith('_file') and k not in ['device', 'mode', 'save_dir', 'load_name', 'save_name']:
+            args[k] = loaded_args[k]
+    
+    eval_input_files = {
+            'txt': args['txt_file'],
+            'label': args['label_file']
+            }
+
+
+    batches = TokenizationDataset(args, input_files=eval_input_files, vocab=vocab, evaluation=True, dictionary=trainer.dictionary)
+
+    oov_count, N, _, _ = output_predictions(args['conll_file'], trainer, batches, vocab, mwt_dict, args['max_seqlen'])
+
+    logger.info("OOV rate: {:6.3f}% ({:6d}/{:6d})".format(oov_count / N * 100, oov_count, N))
+
+
+if __name__ == '__main__':
+    main()

stanza/stanza/models/wl_coref.py

@@ -0,0 +1,226 @@
+"""
+Runs experiments with CorefModel.
+
+Try 'python wl_coref.py -h' for more details.
+
+Code based on
+
+https://github.com/KarelDO/wl-coref/tree/master
+https://arxiv.org/abs/2310.06165
+
+This was a fork of
+
+https://github.com/vdobrovolskii/wl-coref
+https://aclanthology.org/2021.emnlp-main.605/
+
+If you use Stanza's coref module in your work, please cite the following:
+
+@misc{doosterlinck2023cawcoref,
+  title={CAW-coref: Conjunction-Aware Word-level Coreference Resolution},
+  author={Karel D'Oosterlinck and Semere Kiros Bitew and Brandon Papineau and Christopher Potts and Thomas Demeester and Chris Develder},
+  year={2023},
+  eprint={2310.06165},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL},
+  url = "https://arxiv.org/abs/2310.06165",
+}
+
+@inproceedings{dobrovolskii-2021-word,
+  title = "Word-Level Coreference Resolution",
+  author = "Dobrovolskii, Vladimir",
+  booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
+  month = nov,
+  year = "2021",
+  address = "Online and Punta Cana, Dominican Republic",
+  publisher = "Association for Computational Linguistics",
+  url = "https://aclanthology.org/2021.emnlp-main.605",
+  pages = "7670--7675"
+}
+"""
+
+import argparse
+from contextlib import contextmanager
+import datetime
+import logging
+import os
+import random
+import sys
+import dataclasses
+import time
+
+
+import numpy as np  # type: ignore
+import torch        # type: ignore
+
+from stanza.models.common.utils import set_random_seed
+from stanza.models.coref.model import CorefModel
+
+
+logger = logging.getLogger('stanza')
+
+@contextmanager
+def output_running_time():
+    """ Prints the time elapsed in the context """
+    start = int(time.time())
+    try:
+        yield
+    finally:
+        end = int(time.time())
+        delta = datetime.timedelta(seconds=end - start)
+        logger.info(f"Total running time: {delta}")
+
+
+def deterministic() -> None:
+    torch.backends.cudnn.deterministic = True   # type: ignore
+    torch.backends.cudnn.benchmark = False      # type: ignore
+
+
+if __name__ == "__main__":
+    argparser = argparse.ArgumentParser()
+    argparser.add_argument("mode", choices=("train", "eval"))
+    argparser.add_argument("experiment")
+    argparser.add_argument("--config-file", default="config.toml")
+    argparser.add_argument("--data-split", choices=("train", "dev", "test"),
+                           default="test",
+                           help="Data split to be used for evaluation."
+                                " Defaults to 'test'."
+                                " Ignored in 'train' mode.")
+    argparser.add_argument("--batch-size", type=int,
+                           help="Adjust to override the config value of anaphoricity "
+                                "batch size if you are experiencing out-of-memory "
+                                "issues")
+    argparser.add_argument("--disable_singletons", action="store_true",
+                           help="don't predict singletons")
+    argparser.add_argument("--full_pairwise", action="store_true",
+                           help="use speaker and document embeddings")
+    argparser.add_argument("--hidden_size", type=int,
+                           help="Adjust the anaphoricity scorer hidden size")
+    argparser.add_argument("--rough_k", type=int,
+                           help="Adjust the number of dummies to keep")
+    argparser.add_argument("--n_hidden_layers", type=int,
+                           help="Adjust the anaphoricity scorer hidden layers")
+    argparser.add_argument("--dummy_mix", type=float,
+                           help="Adjust the dummy mix")
+    argparser.add_argument("--bert_finetune_begin_epoch", type=float,
+                           help="Adjust the bert finetune begin epoch")
+    argparser.add_argument("--warm_start", action="store_true",
+                           help="If set, the training will resume from the"
+                                " last checkpoint saved if any. Ignored in"
+                                " evaluation modes."
+                                " Incompatible with '--weights'.")
+    argparser.add_argument("--weights",
+                           help="Path to file with weights to load."
+                                " If not supplied, in 'eval' mode the latest"
+                                " weights of the experiment will be loaded;"
+                                " in 'train' mode no weights will be loaded.")
+    argparser.add_argument("--word-level", action="store_true",
+                           help="If set, output word-level conll-formatted"
+                                " files in evaluation modes. Ignored in"
+                                " 'train' mode.")
+    argparser.add_argument("--learning_rate", default=None, type=float,
+                           help="If set, update the learning rate for the model")
+    argparser.add_argument("--bert_learning_rate", default=None, type=float,
+                           help="If set, update the learning rate for the transformer")
+    argparser.add_argument("--save_dir", default=None,
+                           help="If set, update the save directory for writing models")
+    argparser.add_argument("--save_name", default=None,
+                           help="If set, update the save name for writing models (otherwise, section name)")
+    argparser.add_argument("--score_lang", default=None,
+                           help="only score a particular language for eval")
+    argparser.add_argument("--log_norms", action="store_true", default=None,
+                           help="If set, log all of the trainable norms each epoch.  Very noisy!")
+    argparser.add_argument("--seed", type=int, default=2020,
+                           help="Random seed to set")
+
+    argparser.add_argument("--train_data", default=None, help="File to use for train data")
+    argparser.add_argument("--dev_data", default=None, help="File to use for dev data")
+    argparser.add_argument("--test_data", default=None, help="File to use for test data")
+
+    argparser.add_argument('--wandb', action='store_true', help='Start a wandb session and write the results of training.  Only applies to training.  Use --wandb_name instead to specify a name', default=False)
+    argparser.add_argument('--wandb_name', default=None, help='Name of a wandb session to start when training.  Will default to the dataset short name')
+
+    args = argparser.parse_args()
+
+    if args.warm_start and args.weights is not None:
+        raise ValueError("The following options are incompatible: '--warm_start' and '--weights'")
+
+    set_random_seed(args.seed)
+    deterministic()
+    config = CorefModel._load_config(args.config_file, args.experiment)
+    if args.batch_size:
+        config.a_scoring_batch_size = args.batch_size
+    if args.hidden_size:
+        config.hidden_size = args.hidden_size
+    if args.n_hidden_layers:
+        config.n_hidden_layers = args.n_hidden_layers
+    if args.learning_rate is not None:
+        config.learning_rate = args.learning_rate
+    if args.bert_learning_rate is not None:
+        config.bert_learning_rate = args.bert_learning_rate
+    if args.bert_finetune_begin_epoch is not None:
+        config.bert_finetune_begin_epoch = args.bert_finetune_begin_epoch
+    if args.dummy_mix is not None:
+        config.dummy_mix = args.dummy_mix
+
+    if args.save_dir is not None:
+        config.save_dir = args.save_dir
+    if args.save_name:
+        config.save_name = args.save_name
+    else:
+        config.save_name = args.experiment
+
+    if args.rough_k is not None:
+        config.rough_k = args.rough_k
+    if args.log_norms is not None:
+        config.log_norms = args.log_norms
+    if args.full_pairwise:
+        config.full_pairwise = args.full_pairwise
+    if args.disable_singletons:
+        config.singletons = False
+    if args.train_data:
+        config.train_data = args.train_data
+    if args.dev_data:
+        config.dev_data = args.dev_data
+    if args.test_data:
+        config.test_data = args.test_data
+
+    # if wandb, generate wandb configuration 
+    if args.mode == "train":
+        if args.wandb:
+            import wandb
+            wandb_name = args.wandb_name if args.wandb_name else f"wl_coref_{args.experiment}"
+            wandb.init(name=wandb_name, config=dataclasses.asdict(config), project="stanza")
+            wandb.run.define_metric('train_c_loss', summary='min')
+            wandb.run.define_metric('train_s_loss', summary='min')
+            wandb.run.define_metric('dev_score', summary='max')
+
+        model = CorefModel(config=config)
+        if args.weights is not None or args.warm_start:
+            model.load_weights(path=args.weights, map_location="cpu",
+                               noexception=args.warm_start)
+        with output_running_time():
+            model.train(args.wandb)
+    else:
+        config_update = {
+            'log_norms': args.log_norms if args.log_norms is not None else False
+        }
+        if args.test_data:
+            config_update['test_data'] = args.test_data
+
+        if args.weights is None and config.save_name is not None:
+            args.weights = config.save_name
+        if not os.path.exists(args.weights) and os.path.exists(args.weights + ".pt"):
+            args.weights = args.weights + ".pt"
+        elif not os.path.exists(args.weights) and config.save_dir and os.path.exists(os.path.join(config.save_dir, args.weights)):
+            args.weights = os.path.join(config.save_dir, args.weights)
+        elif not os.path.exists(args.weights) and config.save_dir and os.path.exists(os.path.join(config.save_dir, args.weights + ".pt")):
+            args.weights = os.path.join(config.save_dir, args.weights + ".pt")
+        model = CorefModel.load_model(path=args.weights, map_location="cpu",
+                                      ignore={"bert_optimizer", "general_optimizer",
+                                              "bert_scheduler", "general_scheduler"},
+                                      config_update=config_update)
+        results = model.evaluate(data_split=args.data_split,
+                                 word_level_conll=args.word_level, 
+                                 eval_lang=args.score_lang)
+        # logger.info(("mean loss", "))
+        print("\t".join([str(round(i, 3)) for i in results]))

stanza/stanza/pipeline/constituency_processor.py

@@ -0,0 +1,81 @@
+"""
+Processor that attaches a constituency tree to a sentence
+"""
+
+from stanza.models.constituency.trainer import Trainer
+
+from stanza.models.common import doc
+from stanza.models.common.utils import sort_with_indices, unsort
+from stanza.utils.get_tqdm import get_tqdm
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+
+tqdm = get_tqdm()
+
+@register_processor(CONSTITUENCY)
+class ConstituencyProcessor(UDProcessor):
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([CONSTITUENCY])
+    # set of processor requirements for this processor
+    REQUIRES_DEFAULT = set([TOKENIZE, POS])
+
+    # default batch size, measured in sentences
+    DEFAULT_BATCH_SIZE = 50
+
+    def _set_up_requires(self):
+        self._pretagged = self._config.get('pretagged')
+        if self._pretagged:
+            self._requires = set()
+        else:
+            self._requires = self.__class__.REQUIRES_DEFAULT
+
+    def _set_up_model(self, config, pipeline, device):
+        # set up model
+        # pretrain and charlm paths are args from the config
+        # bert (if used) will be chosen from the model save file
+        args = {
+            "wordvec_pretrain_file": config.get('pretrain_path', None),
+            "charlm_forward_file": config.get('forward_charlm_path', None),
+            "charlm_backward_file": config.get('backward_charlm_path', None),
+            "device": device,
+        }
+        trainer = Trainer.load(filename=config['model_path'],
+                               args=args,
+                               foundation_cache=pipeline.foundation_cache)
+        self._trainer = trainer
+        self._model = trainer.model
+        self._model.eval()
+        # batch size counted as sentences
+        self._batch_size = int(config.get('batch_size', ConstituencyProcessor.DEFAULT_BATCH_SIZE))
+        self._tqdm = 'tqdm' in config and config['tqdm']
+
+    def _set_up_final_config(self, config):
+        loaded_args = self._model.args
+        loaded_args = {k: v for k, v in loaded_args.items() if not UDProcessor.filter_out_option(k)}
+        loaded_args.update(config)
+        self._config = loaded_args
+
+    def process(self, document):
+        sentences = document.sentences
+
+        if self._model.uses_xpos():
+            words = [[(w.text, w.xpos) for w in s.words] for s in sentences]
+        else:
+            words = [[(w.text, w.upos) for w in s.words] for s in sentences]
+        words, original_indices = sort_with_indices(words, key=len, reverse=True)
+        if self._tqdm:
+            words = tqdm(words)
+
+        trees = self._model.parse_tagged_words(words, self._batch_size)
+        trees = unsort(trees, original_indices)
+        document.set(CONSTITUENCY, trees, to_sentence=True)
+        return document
+
+    def get_constituents(self):
+        """
+        Return a set of the constituents known by this model
+
+        For a pipeline, this can be queried with
+          pipeline.processors["constituency"].get_constituents()
+        """
+        return set(self._model.constituents)

stanza/stanza/pipeline/core.py

@@ -0,0 +1,509 @@
+"""
+Pipeline that runs tokenize,mwt,pos,lemma,depparse
+"""
+
+import argparse
+import collections
+from enum import Enum
+import io
+import itertools
+import sys
+import logging
+import json
+import os
+
+from stanza.pipeline._constants import *
+from stanza.models.common.constant import langcode_to_lang
+from stanza.models.common.doc import Document
+from stanza.models.common.foundation_cache import FoundationCache
+from stanza.models.common.utils import default_device
+from stanza.pipeline.processor import Processor, ProcessorRequirementsException
+from stanza.pipeline.registry import NAME_TO_PROCESSOR_CLASS, PIPELINE_NAMES, PROCESSOR_VARIANTS
+from stanza.pipeline.langid_processor import LangIDProcessor
+from stanza.pipeline.tokenize_processor import TokenizeProcessor
+from stanza.pipeline.mwt_processor import MWTProcessor
+from stanza.pipeline.pos_processor import POSProcessor
+from stanza.pipeline.lemma_processor import LemmaProcessor
+from stanza.pipeline.constituency_processor import ConstituencyProcessor
+from stanza.pipeline.coref_processor import CorefProcessor
+from stanza.pipeline.depparse_processor import DepparseProcessor
+from stanza.pipeline.sentiment_processor import SentimentProcessor
+from stanza.pipeline.ner_processor import NERProcessor
+from stanza.resources.common import DEFAULT_MODEL_DIR, DEFAULT_RESOURCES_URL, DEFAULT_RESOURCES_VERSION, ModelSpecification, add_dependencies, add_mwt, download_models, download_resources_json, flatten_processor_list, load_resources_json, maintain_processor_list, process_pipeline_parameters, set_logging_level, sort_processors
+from stanza.resources.default_packages import PACKAGES
+from stanza.utils.conll import CoNLL, CoNLLError
+from stanza.utils.helper_func import make_table
+
+logger = logging.getLogger('stanza')
+
+class DownloadMethod(Enum):
+    """
+    Determines a couple options on how to download resources for the pipeline.
+
+    NONE will not download anything, including HF transformers, probably resulting in failure if the resources aren't already in place.
+    REUSE_RESOURCES will reuse the existing resources.json and models, but will download any missing models.
+    DOWNLOAD_RESOURCES will download a new resources.json and will overwrite any out of date models.
+    """
+    NONE = 1
+    REUSE_RESOURCES = 2
+    DOWNLOAD_RESOURCES = 3
+
+class LanguageNotDownloadedError(FileNotFoundError):
+    def __init__(self, lang, lang_dir, model_path):
+        super().__init__(f'Could not find the model file {model_path}.  The expected model directory {lang_dir} is missing.  Perhaps you need to run stanza.download("{lang}")')
+        self.lang = lang
+        self.lang_dir = lang_dir
+        self.model_path = model_path
+
+class UnsupportedProcessorError(FileNotFoundError):
+    def __init__(self, processor, lang):
+        super().__init__(f'Processor {processor} is not known for language {lang}.  If you have created your own model, please specify the {processor}_model_path parameter when creating the pipeline.')
+        self.processor = processor
+        self.lang = lang
+
+class IllegalPackageError(ValueError):
+    def __init__(self, msg):
+        super().__init__(msg)
+
+class PipelineRequirementsException(Exception):
+    """
+    Exception indicating one or more requirements failures while attempting to build a pipeline.
+    Contains a ProcessorRequirementsException list.
+    """
+
+    def __init__(self, processor_req_fails):
+        self._processor_req_fails = processor_req_fails
+        self.build_message()
+
+    @property
+    def processor_req_fails(self):
+        return self._processor_req_fails
+
+    def build_message(self):
+        err_msg = io.StringIO()
+        print(*[req_fail.message for req_fail in self.processor_req_fails], sep='\n', file=err_msg)
+        self.message = '\n\n' + err_msg.getvalue()
+
+    def __str__(self):
+        return self.message
+
+def build_default_config_option(model_specs):
+    """
+    Build a config option for a couple situations: lemma=identity, processor is a variant
+
+    Returns the option name and value
+
+    Refactored from build_default_config so that we can reuse it when
+    downloading all models
+    """
+    # handle case when processor variants are used
+    if any(model_spec.package in PROCESSOR_VARIANTS[model_spec.processor] for model_spec in model_specs):
+        if len(model_specs) > 1:
+            raise IllegalPackageError("Variant processor selected for {}, but multiple packages requested".format(model_spec.processor))
+        return f"{model_specs[0].processor}_with_{model_specs[0].package}", True
+    # handle case when identity is specified as lemmatizer
+    elif any(model_spec.processor == LEMMA and model_spec.package == 'identity' for model_spec in model_specs):
+        if len(model_specs) > 1:
+            raise IllegalPackageError("Identity processor selected for lemma, but multiple packages requested")
+        return f"{LEMMA}_use_identity", True
+    return None
+
+def filter_variants(model_specs):
+    return [(key, value) for (key, value) in model_specs if build_default_config_option(value) is None]
+
+# given a language and models path, build a default configuration
+def build_default_config(resources, lang, model_dir, load_list):
+    default_config = {}
+    for processor, model_specs in load_list:
+        option = build_default_config_option(model_specs)
+        if option is not None:
+            # if an option is set for the model_specs, keep that option and ignore
+            # the rest of the model spec
+            default_config[option[0]] = option[1]
+            continue
+
+        model_paths = [os.path.join(model_dir, lang, processor, model_spec.package + '.pt') for model_spec in model_specs]
+        dependencies = [model_spec.dependencies for model_spec in model_specs]
+
+        # Special case for NER: load multiple models at once
+        # The pattern will be:
+        #   a list of ner_model_path
+        #   a list of ner_dependencies
+        #     where each item in ner_dependencies is a map
+        #     the map may contain forward_charlm_path, backward_charlm_path, or any other deps
+        # The user will be able to override the defaults using a semicolon separated string
+        # TODO: at least use the same config pattern for all other models
+        if processor == NER:
+            default_config[f"{processor}_model_path"] = model_paths
+            dependency_paths = []
+            for dependency_block in dependencies:
+                if not dependency_block:
+                    dependency_paths.append({})
+                    continue
+                dependency_paths.append({})
+                for dependency in dependency_block:
+                    dep_processor, dep_model = dependency
+                    dependency_paths[-1][f"{dep_processor}_path"] = os.path.join(model_dir, lang, dep_processor, dep_model + '.pt')
+            default_config[f"{processor}_dependencies"] = dependency_paths
+            continue
+
+        if len(model_specs) > 1:
+            raise IllegalPackageError("Specified multiple packages for {}, which currently only handles one package".format(processor))
+
+        default_config[f"{processor}_model_path"] = model_paths[0]
+        if not dependencies[0]: continue
+        for dependency in dependencies[0]:
+            dep_processor, dep_model = dependency
+            default_config[f"{processor}_{dep_processor}_path"] = os.path.join(
+                model_dir, lang, dep_processor, dep_model + '.pt'
+            )
+
+    return default_config
+
+def normalize_download_method(download_method):
+    """
+    Turn None -> DownloadMethod.NONE, strings to the corresponding enum
+    """
+    if download_method is None:
+        return DownloadMethod.NONE
+    elif isinstance(download_method, str):
+        try:
+            return DownloadMethod[download_method.upper()]
+        except KeyError as e:
+            raise ValueError("Unknown download method %s" % download_method) from e
+    return download_method
+
+class Pipeline:
+
+    def __init__(self,
+                 lang='en',
+                 dir=DEFAULT_MODEL_DIR,
+                 package='default',
+                 processors={},
+                 logging_level=None,
+                 verbose=None,
+                 use_gpu=None,
+                 model_dir=None,
+                 download_method=DownloadMethod.DOWNLOAD_RESOURCES,
+                 resources_url=DEFAULT_RESOURCES_URL,
+                 resources_branch=None,
+                 resources_version=DEFAULT_RESOURCES_VERSION,
+                 resources_filepath=None,
+                 proxies=None,
+                 foundation_cache=None,
+                 device=None,
+                 allow_unknown_language=False,
+                 **kwargs):
+        self.lang, self.dir, self.kwargs = lang, dir, kwargs
+        if model_dir is not None and dir == DEFAULT_MODEL_DIR:
+            self.dir = model_dir
+
+        # set global logging level
+        set_logging_level(logging_level, verbose)
+
+        self.download_method = normalize_download_method(download_method)
+        if (self.download_method is DownloadMethod.DOWNLOAD_RESOURCES or
+            (self.download_method is DownloadMethod.REUSE_RESOURCES and not os.path.exists(os.path.join(self.dir, "resources.json")))):
+            logger.info("Checking for updates to resources.json in case models have been updated.  Note: this behavior can be turned off with download_method=None or download_method=DownloadMethod.REUSE_RESOURCES")
+            download_resources_json(self.dir,
+                                    resources_url=resources_url,
+                                    resources_branch=resources_branch,
+                                    resources_version=resources_version,
+                                    resources_filepath=resources_filepath,
+                                    proxies=proxies)
+
+        # processors can use this to save on the effort of loading
+        # large sub-models, such as pretrained embeddings, bert, etc
+        if foundation_cache is None:
+            self.foundation_cache = FoundationCache(local_files_only=(self.download_method is DownloadMethod.NONE))
+        else:
+            self.foundation_cache = FoundationCache(foundation_cache, local_files_only=(self.download_method is DownloadMethod.NONE))
+
+        # process different pipeline parameters
+        lang, self.dir, package, processors = process_pipeline_parameters(lang, self.dir, package, processors)
+
+        # Load resources.json to obtain latest packages.
+        logger.debug('Loading resource file...')
+        resources = load_resources_json(self.dir, resources_filepath)
+        if lang in resources:
+            if 'alias' in resources[lang]:
+                logger.info(f'"{lang}" is an alias for "{resources[lang]["alias"]}"')
+                lang = resources[lang]['alias']
+            lang_name = resources[lang]['lang_name'] if 'lang_name' in resources[lang] else ''
+        elif allow_unknown_language:
+            logger.warning("Trying to create pipeline for unsupported language: %s", lang)
+            lang_name = langcode_to_lang(lang)
+        else:
+            logger.warning("Unsupported language: %s  If trying to add a new language, consider using allow_unknown_language=True", lang)
+            lang_name = langcode_to_lang(lang)
+
+        # Maintain load list
+        if lang in resources:
+            self.load_list = maintain_processor_list(resources, lang, package, processors, maybe_add_mwt=(not kwargs.get("tokenize_pretokenized")))
+            self.load_list = add_dependencies(resources, lang, self.load_list)
+            if self.download_method is not DownloadMethod.NONE:
+                # skip processors which aren't downloaded from our collection
+                download_list = [x for x in self.load_list if x[0] in resources.get(lang, {})]
+                # skip variants
+                download_list = filter_variants(download_list)
+                # gather up the model list...
+                download_list = flatten_processor_list(download_list)
+                # download_models will skip models we already have
+                download_models(download_list,
+                                resources=resources,
+                                lang=lang,
+                                model_dir=self.dir,
+                                resources_version=resources_version,
+                                proxies=proxies,
+                                log_info=False)
+        elif allow_unknown_language:
+            self.load_list = [(proc, [ModelSpecification(processor=proc, package='default', dependencies=None)])
+                              for proc in list(processors.keys())]
+        else:
+            self.load_list = []
+        self.load_list = self.update_kwargs(kwargs, self.load_list)
+        if len(self.load_list) == 0:
+            if lang not in resources or PACKAGES not in resources[lang]:
+                raise ValueError(f'No processors to load for language {lang}.  Language {lang} is currently unsupported')
+            else:
+                raise ValueError('No processors to load for language {}.  Please check if your language or package is correctly set.'.format(lang))
+        load_table = make_table(['Processor', 'Package'], [(row[0], ";".join(model_spec.package for model_spec in row[1])) for row in self.load_list])
+        logger.info(f'Loading these models for language: {lang} ({lang_name}):\n{load_table}')
+
+        self.config = build_default_config(resources, lang, self.dir, self.load_list)
+        self.config.update(kwargs)
+
+        # Load processors
+        self.processors = {}
+
+        # configs that are the same for all processors
+        pipeline_level_configs = {'lang': lang, 'mode': 'predict'}
+
+        if device is None:
+            if use_gpu is None or use_gpu == True:
+                device = default_device()
+            else:
+                device = 'cpu'
+            if use_gpu == True and device == 'cpu':
+                logger.warning("GPU requested, but is not available!")
+        self.device = device
+        logger.info("Using device: {}".format(self.device))
+
+        # set up processors
+        pipeline_reqs_exceptions = []
+        for item in self.load_list:
+            processor_name, _ = item
+            logger.info('Loading: ' + processor_name)
+            curr_processor_config = self.filter_config(processor_name, self.config)
+            curr_processor_config.update(pipeline_level_configs)
+            # TODO: this is obviously a hack
+            # a better solution overall would be to make a pretagged version of the pos annotator
+            # and then subsequent modules can use those tags without knowing where those tags came from
+            if "pretagged" in self.config and "pretagged" not in curr_processor_config:
+                curr_processor_config["pretagged"] = self.config["pretagged"]
+            logger.debug('With settings: ')
+            logger.debug(curr_processor_config)
+            try:
+                # try to build processor, throw an exception if there is a requirements issue
+                self.processors[processor_name] = NAME_TO_PROCESSOR_CLASS[processor_name](config=curr_processor_config,
+                                                                                          pipeline=self,
+                                                                                          device=self.device)
+            except ProcessorRequirementsException as e:
+                # if there was a requirements issue, add it to list which will be printed at end
+                pipeline_reqs_exceptions.append(e)
+                # add the broken processor to the loaded processors for the sake of analyzing the validity of the
+                # entire proposed pipeline, but at this point the pipeline will not be built successfully
+                self.processors[processor_name] = e.err_processor
+            except FileNotFoundError as e:
+                # For a FileNotFoundError, we try to guess if there's
+                # a missing model directory or file.  If so, we
+                # suggest the user try to download the models
+                if 'model_path' in curr_processor_config:
+                    model_path = curr_processor_config['model_path']
+                    if e.filename == model_path or (isinstance(model_path, (tuple, list)) and e.filename in model_path):
+                        model_path = e.filename
+                    model_dir, model_name = os.path.split(model_path)
+                    lang_dir = os.path.dirname(model_dir)
+                    if lang_dir and not os.path.exists(lang_dir):
+                        # model files for this language can't be found in the expected directory
+                        raise LanguageNotDownloadedError(lang, lang_dir, model_path) from e
+                    if processor_name not in resources[lang]:
+                        # user asked for a model which doesn't exist for this language?
+                        raise UnsupportedProcessorError(processor_name, lang) from e
+                    if not os.path.exists(model_path):
+                        model_name, _ = os.path.splitext(model_name)
+                        # TODO: before recommending this, check that such a thing exists in resources.json.
+                        # currently that case is handled by ignoring the model, anyway
+                        raise FileNotFoundError('Could not find model file %s, although there are other models downloaded for language %s.  Perhaps you need to download a specific model.  Try: stanza.download(lang="%s",package=None,processors={"%s":"%s"})' % (model_path, lang, lang, processor_name, model_name)) from e
+
+                # if we couldn't find a more suitable description of the
+                # FileNotFoundError, just raise the old error
+                raise
+
+        # if there are any processor exceptions, throw an exception to indicate pipeline build failure
+        if pipeline_reqs_exceptions:
+            logger.info('\n')
+            raise PipelineRequirementsException(pipeline_reqs_exceptions)
+
+        logger.info("Done loading processors!")
+
+    @staticmethod
+    def update_kwargs(kwargs, processor_list):
+        processor_dict = {processor: [{'package': model_spec.package, 'dependencies': model_spec.dependencies} for model_spec in model_specs]
+                          for (processor, model_specs) in processor_list}
+        for key, value in kwargs.items():
+            pieces = key.split('_', 1)
+            if len(pieces) == 1:
+                continue
+            k, v = pieces
+            if v == 'model_path':
+                package = value if len(value) < 25 else value[:10]+ '...' + value[-10:]
+                original_spec = processor_dict.get(k, [])
+                if len(original_spec) > 0:
+                    dependencies = original_spec[0].get('dependencies')
+                else:
+                    dependencies = None
+                processor_dict[k] = [{'package': package, 'dependencies': dependencies}]
+        processor_list = [(processor, [ModelSpecification(processor=processor, package=model_spec['package'], dependencies=model_spec['dependencies']) for model_spec in processor_dict[processor]]) for processor in processor_dict]
+        processor_list = sort_processors(processor_list)
+        return processor_list
+
+    @staticmethod
+    def filter_config(prefix, config_dict):
+        filtered_dict = {}
+        for key in config_dict.keys():
+            pieces = key.split('_', 1)  # split tokenize_pretokenize to tokenize+pretokenize
+            if len(pieces) == 1:
+                continue
+            k, v = pieces
+            if k == prefix:
+                filtered_dict[v] = config_dict[key]
+        return filtered_dict
+
+    @property
+    def loaded_processors(self):
+        """
+        Return all currently loaded processors in execution order.
+        :return: list of Processor instances
+        """
+        return [self.processors[processor_name] for processor_name in PIPELINE_NAMES if self.processors.get(processor_name)]
+
+    def process(self, doc, processors=None):
+        """
+        Run the pipeline
+
+        processors: allow for a list of processors used by this pipeline action
+          can be list, tuple, set, or comma separated string
+          if None, use all the processors this pipeline knows about
+          MWT is added if necessary
+          otherwise, no care is taken to make sure prerequisites are followed...
+            some of the annotators, such as depparse, will check, but others
+            will fail in some unusual manner or just have really bad results
+        """
+        assert any([isinstance(doc, str), isinstance(doc, list),
+                    isinstance(doc, Document)]), 'input should be either str, list or Document'
+
+        # empty bulk process
+        if isinstance(doc, list) and len(doc) == 0:
+            return []
+
+        # determine whether we are in bulk processing mode for multiple documents
+        bulk=(isinstance(doc, list) and len(doc) > 0 and isinstance(doc[0], Document))
+
+        # various options to limit the processors used by this pipeline action
+        if processors is None:
+            processors = PIPELINE_NAMES
+        elif not isinstance(processors, (str, list, tuple, set)):
+            raise ValueError("Cannot process {} as a list of processors to run".format(type(processors)))
+        else:
+            if isinstance(processors, str):
+                processors = {x for x in processors.split(",")}
+            else:
+                processors = set(processors)
+            if TOKENIZE in processors and MWT in self.processors and MWT not in processors:
+                logger.debug("Requested processors for pipeline did not have mwt, but pipeline needs mwt, so mwt is added")
+                processors.add(MWT)
+            processors = [x for x in PIPELINE_NAMES if x in processors]
+
+        for processor_name in processors:
+            if self.processors.get(processor_name):
+                process = self.processors[processor_name].bulk_process if bulk else self.processors[processor_name].process
+                doc = process(doc)
+        return doc
+
+    def bulk_process(self, docs, *args, **kwargs):
+        """
+        Run the pipeline in bulk processing mode
+
+        Expects a list of str or a list of Docs
+        """
+        # Wrap each text as a Document unless it is already such a document
+        docs = [doc if isinstance(doc, Document) else Document([], text=doc) for doc in docs]
+        return self.process(docs, *args, **kwargs)
+
+    def stream(self, docs, batch_size=50, *args, **kwargs):
+        """
+        Go through an iterator of documents in batches, yield processed documents
+
+        sentence indices will be counted across the entire iterator
+        """
+        if not isinstance(docs, collections.abc.Iterator):
+            docs = iter(docs)
+        def next_batch():
+            batch = []
+            for _ in range(batch_size):
+                try:
+                    next_doc = next(docs)
+                    batch.append(next_doc)
+                except StopIteration:
+                    return batch
+            return batch
+
+        sentence_start_index = 0
+        batch = next_batch()
+        while batch:
+            batch = self.bulk_process(batch, *args, **kwargs)
+            for doc in batch:
+                doc.reindex_sentences(sentence_start_index)
+                sentence_start_index += len(doc.sentences)
+                yield doc
+            batch = next_batch()
+
+    def __str__(self):
+        """
+        Assemble the processors in order to make a simple description of the pipeline
+        """
+        processors = ["%s=%s" % (x, str(self.processors[x])) for x in PIPELINE_NAMES if x in self.processors]
+        return "<Pipeline: %s>" % ", ".join(processors)
+
+    def __call__(self, doc, processors=None):
+        return self.process(doc, processors)
+
+def main():
+    # TODO: can add a bunch more arguments
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--lang', type=str, default='en', help='Language of the pipeline to use')
+    parser.add_argument('--input_file', type=str, required=True, help='Input file to read')
+    parser.add_argument('--processors', type=str, default='tokenize,pos,lemma,depparse', help='Processors to use')
+    args, extra_args = parser.parse_known_args()
+
+    try:
+        doc = CoNLL.conll2doc(args.input_file)
+        extra_args = {
+            "tokenize_pretokenized": True
+        }
+    except CoNLLError:
+        logger.debug("Input file %s does not appear to be a conllu file.  Will read it as a text file")
+        with open(args.input_file, encoding="utf-8") as fin:
+            doc = fin.read()
+        extra_args = {}
+
+    pipe = Pipeline(args.lang, processors=args.processors, **extra_args)
+
+    doc = pipe(doc)
+
+    print("{:C}".format(doc))
+
+
+if __name__ == '__main__':
+    main()

stanza/stanza/pipeline/coref_processor.py

@@ -0,0 +1,154 @@
+"""
+Processor that attaches coref annotations to a document
+"""
+
+from stanza.models.common.utils import misc_to_space_after
+from stanza.models.coref.coref_chain import CorefMention, CorefChain
+
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+
+def extract_text(document, sent_id, start_word, end_word):
+    sentence = document.sentences[sent_id]
+    tokens = []
+
+    # the coref model indexes the words from 0,
+    # whereas the ids we are looking at on the tokens start from 1
+    # here we will switch to ID space
+    start_word = start_word + 1
+    end_word = end_word + 1
+
+    # For each position between start and end word:
+    # If a word is part of an MWT, and the entire token
+    # is inside the range, we use that Token's text for that span
+    # This will let us easily handle words which are split into pieces
+    # Otherwise, we only take the text of the word itself
+    next_idx = start_word
+    while next_idx < end_word:
+        word = sentence.words[next_idx-1]
+        parent_token = word.parent
+        if isinstance(parent_token.id, int) or len(parent_token.id) == 1:
+            tokens.append(parent_token)
+            next_idx += 1
+        elif parent_token.id[0] >= start_word and parent_token.id[1] < end_word:
+            tokens.append(parent_token)
+            next_idx = parent_token.id[1] + 1
+        else:
+            tokens.append(word)
+            next_idx += 1
+
+    # We use the SpaceAfter or SpacesAfter attribute on each Word or Token
+    # we chose in the above loop to separate the text pieces
+    text = []
+    for token in tokens:
+        text.append(token.text)
+        text.append(misc_to_space_after(token.misc))
+    # the last token space_after will be discarded
+    # so that we don't have stray WS at the end of the mention text
+    text = text[:-1]
+    return "".join(text)
+
+
+@register_processor(COREF)
+class CorefProcessor(UDProcessor):
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([COREF])
+    # set of processor requirements for this processor
+    REQUIRES_DEFAULT = set([TOKENIZE])
+
+    def _set_up_model(self, config, pipeline, device):
+        try:
+            from stanza.models.coref.model import CorefModel
+        except ImportError:
+            raise ImportError("Please install the transformers and peft libraries before using coref! Try `pip install -e .[transformers]`.")
+
+        # set up model
+        # currently, the model has everything packaged in it
+        # (except its config)
+        # TODO: separate any pretrains if possible
+        # TODO: add device parameter to the load mechanism
+        config_update = {'log_norms': False,
+                         'device': device}
+        model = CorefModel.load_model(path=config['model_path'],
+                                      ignore={"bert_optimizer", "general_optimizer",
+                                              "bert_scheduler", "general_scheduler"},
+                                      config_update=config_update,
+                                      foundation_cache=pipeline.foundation_cache)
+        if config.get('batch_size', None):
+            model.config.a_scoring_batch_size = int(config['batch_size'])
+        model.training = False
+
+        self._model = model
+
+    def process(self, document):
+        sentences = document.sentences
+
+        cased_words = []
+        sent_ids = []
+        word_pos = []
+        for sent_idx, sentence in enumerate(sentences):
+            for word_idx, word in enumerate(sentence.words):
+                cased_words.append(word.text)
+                sent_ids.append(sent_idx)
+                word_pos.append(word_idx)
+
+        coref_input = {
+            "document_id": "wb_doc_1",
+            "cased_words": cased_words,
+            "sent_id": sent_ids
+        }
+        coref_input = self._model.build_doc(coref_input)
+        results = self._model.run(coref_input)
+        clusters = []
+        for span_cluster in results.span_clusters:
+            if len(span_cluster) == 0:
+                continue
+            span_cluster = sorted(span_cluster)
+
+            for span in span_cluster:
+                # check there are no sentence crossings before
+                # manipulating the spans, since we will expect it to
+                # be this way for multiple usages of the spans
+                sent_id = sent_ids[span[0]]
+                if sent_ids[span[1]-1] != sent_id:
+                    raise ValueError("The coref model predicted a span that crossed two sentences!  Please send this example to us on our github")
+
+            # treat the longest span as the representative
+            # break ties using the first one
+            # IF there is the POS processor, and it adds upos tags
+            # to the sentence, ties are broken first by maximum
+            # number of UPOS and then earliest in the document
+            max_len = 0
+            best_span = None
+            max_propn = 0
+            for span_idx, span in enumerate(span_cluster):
+                sent_id = sent_ids[span[0]]
+                sentence = sentences[sent_id]
+                start_word = word_pos[span[0]]
+                # fiddle -1 / +1 so as to avoid problems with coref
+                # clusters that end at exactly the end of a document
+                end_word = word_pos[span[1]-1] + 1
+                # very UD specific test for most number of proper nouns in a mention
+                # will do nothing if POS is not active (they will all be None)
+                num_propn = sum(word.pos == 'PROPN' for word in sentence.words[start_word:end_word])
+
+                if ((span[1] - span[0] > max_len) or
+                    span[1] - span[0] == max_len and num_propn > max_propn):
+                    max_len = span[1] - span[0]
+                    best_span = span_idx
+                    max_propn = num_propn
+
+            mentions = []
+            for span in span_cluster:
+                sent_id = sent_ids[span[0]]
+                start_word = word_pos[span[0]]
+                end_word = word_pos[span[1]-1] + 1
+                mentions.append(CorefMention(sent_id, start_word, end_word))
+            representative = mentions[best_span]
+            representative_text = extract_text(document, representative.sentence, representative.start_word, representative.end_word)
+
+            chain = CorefChain(len(clusters), mentions, representative_text, best_span)
+            clusters.append(chain)
+
+        document.coref = clusters
+        return document

stanza/stanza/pipeline/depparse_processor.py

@@ -0,0 +1,78 @@
+"""
+Processor for performing dependency parsing
+"""
+
+import torch
+
+from stanza.models.common import doc
+from stanza.models.common.utils import unsort
+from stanza.models.common.vocab import VOCAB_PREFIX
+from stanza.models.depparse.data import DataLoader
+from stanza.models.depparse.trainer import Trainer
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+
+# these imports trigger the "register_variant" decorations
+from stanza.pipeline.external.corenlp_converter_depparse import ConverterDepparse
+
+DEFAULT_SEPARATE_BATCH=150
+
+@register_processor(name=DEPPARSE)
+class DepparseProcessor(UDProcessor):
+
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([DEPPARSE])
+    # set of processor requirements for this processor
+    REQUIRES_DEFAULT = set([TOKENIZE, POS, LEMMA])
+
+    def __init__(self, config, pipeline, device):
+        self._pretagged = None
+        super().__init__(config, pipeline, device)
+
+    def _set_up_requires(self):
+        self._pretagged = self._config.get('pretagged')
+        if self._pretagged:
+            self._requires = set()
+        else:
+            self._requires = self.__class__.REQUIRES_DEFAULT
+
+    def _set_up_model(self, config, pipeline, device):
+        self._pretrain = pipeline.foundation_cache.load_pretrain(config['pretrain_path']) if 'pretrain_path' in config else None
+        args = {'charlm_forward_file': config.get('forward_charlm_path', None),
+                'charlm_backward_file': config.get('backward_charlm_path', None)}
+        self._trainer = Trainer(args=args, pretrain=self.pretrain, model_file=config['model_path'], device=device, foundation_cache=pipeline.foundation_cache)
+
+    def get_known_relations(self):
+        """
+        Return a list of relations which this processor can produce
+        """
+        keys = [k for k in self.vocab['deprel']._unit2id.keys() if k not in VOCAB_PREFIX]
+        return keys
+
+    def process(self, document):
+        if hasattr(self, '_variant'):
+            return self._variant.process(document)
+
+        if any(word.upos is None and word.xpos is None for sentence in document.sentences for word in sentence.words):
+            raise ValueError("POS not run before depparse!")
+        try:
+            batch = DataLoader(document, self.config['batch_size'], self.config, self.pretrain, vocab=self.vocab, evaluation=True,
+                               sort_during_eval=self.config.get('sort_during_eval', True),
+                               min_length_to_batch_separately=self.config.get('min_length_to_batch_separately', DEFAULT_SEPARATE_BATCH))
+            with torch.no_grad():
+                preds = []
+                for i, b in enumerate(batch):
+                    preds += self.trainer.predict(b)
+            if batch.data_orig_idx is not None:
+                preds = unsort(preds, batch.data_orig_idx)
+            batch.doc.set((doc.HEAD, doc.DEPREL), [y for x in preds for y in x])
+            # build dependencies based on predictions
+            for sentence in batch.doc.sentences:
+                sentence.build_dependencies()
+            return batch.doc
+        except RuntimeError as e:
+            if str(e).startswith("CUDA out of memory. Tried to allocate"):
+                new_message = str(e) + " ... You may be able to compensate for this by separating long sentences into their own batch with a parameter such as depparse_min_length_to_batch_separately=150 or by limiting the overall batch size with depparse_batch_size=400."
+                raise RuntimeError(new_message) from e
+            else:
+                raise

stanza/stanza/pipeline/langid_processor.py

@@ -0,0 +1,127 @@
+"""
+Processor for determining language of text.
+"""
+
+import emoji
+import re
+import stanza
+import torch
+
+from stanza.models.common.doc import Document
+from stanza.models.langid.model import LangIDBiLSTM
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+
+
+@register_processor(name=LANGID)
+class LangIDProcessor(UDProcessor):
+    """
+    Class for detecting language of text.
+    """
+
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([LANGID])
+
+    # set of processor requirements for this processor
+    REQUIRES_DEFAULT = set([])
+
+    # default max sequence length
+    MAX_SEQ_LENGTH_DEFAULT = 1000
+
+    def _set_up_model(self, config, pipeline, device):
+        batch_size = config.get("batch_size", 64)
+        self._model = LangIDBiLSTM.load(path=config["model_path"], device=device,
+                                        batch_size=batch_size, lang_subset=config.get("lang_subset"))
+        self._char_index = self._model.char_to_idx
+        self._clean_text = config.get("clean_text")
+
+    def _text_to_tensor(self, docs):
+        """
+        Map list of strings to batch tensor. Assumed all docs are same length.
+        """
+
+        device = next(self._model.parameters()).device
+        all_docs = []
+        for doc in docs:
+            doc_chars = [self._char_index.get(c, self._char_index["UNK"]) for c in list(doc)]
+            all_docs.append(doc_chars)
+        return torch.tensor(all_docs, device=device, dtype=torch.long)
+
+    def _id_langs(self, batch_tensor):
+        """
+        Identify languages for each sequence in a batch tensor
+        """
+        predictions = self._model.prediction_scores(batch_tensor)
+        prediction_labels = [self._model.idx_to_tag[prediction] for prediction in predictions]
+
+        return prediction_labels
+
+    # regexes for cleaning text
+    http_regex = re.compile(r"https?:\/\/t\.co/[a-zA-Z0-9]+")
+    handle_regex = re.compile("@[a-zA-Z0-9_]+")
+    hashtag_regex = re.compile("#[a-zA-Z]+")
+    punctuation_regex = re.compile("[!.]+")
+    all_regexes = [http_regex, handle_regex, hashtag_regex, punctuation_regex]
+
+    @staticmethod
+    def clean_text(text):
+        """
+        Process text to improve language id performance. Main emphasis is on tweets, this method removes shortened
+        urls, hashtags, handles, and punctuation and emoji.
+        """
+
+        for regex in LangIDProcessor.all_regexes:
+            text = regex.sub(" ", text)
+
+        text = emoji.emojize(text)
+        text = emoji.replace_emoji(text, replace=' ')
+
+        if text.strip():
+            text = text.strip()
+
+        return text
+
+    def _process_list(self, docs):
+        """
+        Identify language of list of strings or Documents
+        """
+
+        if len(docs) == 0:
+            # TO DO: what standard do we want for bad input, such as empty list?
+            # TO DO: more handling of bad input
+            return
+
+        if isinstance(docs[0], str):
+            docs = [Document([], text) for text in docs]
+
+        docs_by_length = {}
+        for doc in docs:
+            text = LangIDProcessor.clean_text(doc.text) if self._clean_text else doc.text
+            doc_length = len(text)
+            if doc_length not in docs_by_length:
+                docs_by_length[doc_length] = []
+            docs_by_length[doc_length].append((doc, text))
+
+        for doc_length in docs_by_length:
+            inputs = [doc[1] for doc in docs_by_length[doc_length]]
+            predictions = self._id_langs(self._text_to_tensor(inputs))
+            for doc, lang in zip(docs_by_length[doc_length], predictions):
+                doc[0].lang = lang
+
+        return docs
+
+    def process(self, doc):
+        """
+        Handle single str or Document
+        """
+
+        wrapped_doc = [doc]
+        return self._process_list(wrapped_doc)[0]
+
+    def bulk_process(self, docs):
+        """
+        Handle list of strings or Documents
+        """
+
+        return self._process_list(docs)
+

stanza/stanza/pipeline/lemma_processor.py

@@ -0,0 +1,126 @@
+"""
+Processor for performing lemmatization
+"""
+
+from itertools import compress
+
+import torch
+
+from stanza.models.common import doc
+from stanza.models.lemma.data import DataLoader
+from stanza.models.lemma.trainer import Trainer
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+
+WORD_TAGS = [doc.TEXT, doc.UPOS]
+
+@register_processor(name=LEMMA)
+class LemmaProcessor(UDProcessor):
+
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([LEMMA])
+    # set of processor requirements for this processor
+    # pos will be added later for non-identity lemmatizerx
+    REQUIRES_DEFAULT = set([TOKENIZE])
+    # default batch size
+    DEFAULT_BATCH_SIZE = 5000
+
+    def __init__(self, config, pipeline, device):
+        # run lemmatizer in identity mode
+        self._use_identity = None
+        self._pretagged = None
+        super().__init__(config, pipeline, device)
+
+    @property
+    def use_identity(self):
+        return self._use_identity
+
+    def _set_up_model(self, config, pipeline, device):
+        if config.get('use_identity') in ['True', True]:
+            self._use_identity = True
+            self._config = config
+            self.config['batch_size'] = LemmaProcessor.DEFAULT_BATCH_SIZE
+        else:
+            # the lemmatizer only looks at one word when making
+            # decisions, not the surrounding context
+            # therefore, we can save some time by remembering what
+            # we did the last time we saw any given word,pos
+            # since a long running program will remember everything
+            # (unless we go back and make it smarter)
+            # we make this an option, not the default
+            # TODO: need to update the cache to skip the contextual lemmatizer
+            self.store_results = config.get('store_results', False)
+            self._use_identity = False
+            args = {'charlm_forward_file': config.get('forward_charlm_path', None),
+                    'charlm_backward_file': config.get('backward_charlm_path', None)}
+            lemma_classifier_args = dict(args)
+            lemma_classifier_args['wordvec_pretrain_file'] = config.get('pretrain_path', None)
+            self._trainer = Trainer(args=args, model_file=config['model_path'], device=device, foundation_cache=pipeline.foundation_cache, lemma_classifier_args=lemma_classifier_args)
+
+    def _set_up_requires(self):
+        self._pretagged = self._config.get('pretagged', None)
+        if self._pretagged:
+            self._requires = set()
+        elif self.config.get('pos') and not self.use_identity:
+            self._requires = LemmaProcessor.REQUIRES_DEFAULT.union(set([POS]))
+        else:
+            self._requires = LemmaProcessor.REQUIRES_DEFAULT
+
+    def process(self, document):
+        if not self.use_identity:
+            batch = DataLoader(document, self.config['batch_size'], self.config, vocab=self.vocab, evaluation=True, expand_unk_vocab=True)
+        else:
+            batch = DataLoader(document, self.config['batch_size'], self.config, evaluation=True, conll_only=True)
+        if self.use_identity:
+            preds = [word.text for sent in batch.doc.sentences for word in sent.words]
+        elif self.config.get('dict_only', False):
+            preds = self.trainer.predict_dict(batch.doc.get([doc.TEXT, doc.UPOS]))
+        else:
+            if self.config.get('ensemble_dict', False):
+                # skip the seq2seq model when we can
+                skip = self.trainer.skip_seq2seq(batch.doc.get([doc.TEXT, doc.UPOS]))
+                # although there is no explicit use of caseless or lemma_caseless in this processor,
+                # it shows up in the config which gets passed to the DataLoader,
+                # possibly affecting its results
+                seq2seq_batch = DataLoader(document, self.config['batch_size'], self.config, vocab=self.vocab,
+                                           evaluation=True, skip=skip, expand_unk_vocab=True)
+            else:
+                seq2seq_batch = batch
+
+            with torch.no_grad():
+                preds = []
+                edits = []
+                for i, b in enumerate(seq2seq_batch):
+                    ps, es = self.trainer.predict(b, self.config['beam_size'], seq2seq_batch.vocab)
+                    preds += ps
+                    if es is not None:
+                        edits += es
+
+            if self.config.get('ensemble_dict', False):
+                word_tags = batch.doc.get(WORD_TAGS)
+                words = [x[0] for x in word_tags]
+                preds = self.trainer.postprocess([x for x, y in zip(words, skip) if not y], preds, edits=edits)
+                if self.store_results:
+                    new_word_tags = compress(word_tags, map(lambda x: not x, skip))
+                    new_predictions = [(x[0], x[1], y) for x, y in zip(new_word_tags, preds)]
+                    self.trainer.train_dict(new_predictions, update_word_dict=False)
+                # expand seq2seq predictions to the same size as all words
+                i = 0
+                preds1 = []
+                for s in skip:
+                    if s:
+                        preds1.append('')
+                    else:
+                        preds1.append(preds[i])
+                        i += 1
+                preds = self.trainer.ensemble(word_tags, preds1)
+            else:
+                preds = self.trainer.postprocess(batch.doc.get([doc.TEXT]), preds, edits=edits)
+
+            if self.trainer.has_contextual_lemmatizers():
+                preds = self.trainer.update_contextual_preds(batch.doc, preds)
+
+        # map empty string lemmas to '_'
+        preds = [max([(len(x), x), (0, '_')])[1] for x in preds]
+        batch.doc.set([doc.LEMMA], preds)
+        return batch.doc

stanza/stanza/pipeline/multilingual.py

@@ -0,0 +1,188 @@
+"""
+Class for running multilingual pipelines
+"""
+
+from collections import OrderedDict
+import copy
+import logging
+from typing import Union
+
+from stanza.models.common.doc import Document
+from stanza.models.common.utils import default_device
+from stanza.pipeline.core import Pipeline, DownloadMethod
+from stanza.pipeline._constants import *
+from stanza.resources.common import DEFAULT_MODEL_DIR, get_language_resources, load_resources_json
+
+logger = logging.getLogger('stanza')
+
+class MultilingualPipeline:
+    """
+    Pipeline for handling multilingual data. Takes in text, detects language, and routes request to pipeline for that
+    language.
+
+    You can specify options to individual language pipelines with the lang_configs field.
+    For example, if you want English pipelines to have NER, but want to turn that off for French, you can do:
+        lang_configs = {"en": {"processors": "tokenize,pos,lemma,depparse,ner"},
+                        "fr": {"processors": "tokenize,pos,lemma,depparse"}}
+        pipeline = MultilingualPipeline(lang_configs=lang_configs)
+
+    You can also pass in a defaultdict created in such a way that it provides default parameters for each language.
+    For example, in order to only get tokenization for each language:
+    (remembering that the Pipeline will automagically add MWT to a language which uses MWT):
+        from collections import defaultdict
+        lang_configs = defaultdict(lambda: dict(processors="tokenize"))
+        pipeline = MultilingualPipeline(lang_configs=lang_configs)
+
+    download_method can be set as in Pipeline to turn off downloading
+      of the .json config or turn off downloading of everything
+    """
+
+    def __init__(self,
+                 model_dir: str = DEFAULT_MODEL_DIR,
+                 lang_id_config: dict = None,
+                 lang_configs: dict = None,
+                 ld_batch_size: int = 64,
+                 max_cache_size: int = 10,
+                 use_gpu: bool = None,
+                 restrict: bool = False,
+                 device: str = None,
+                 download_method: DownloadMethod = DownloadMethod.DOWNLOAD_RESOURCES,
+                 # python 3.6 compatibility - maybe want to update to 3.7 at some point
+                 processors: Union[str, list] = None,
+    ):
+        # set up configs and cache for various language pipelines
+        self.model_dir = model_dir
+        self.lang_id_config = {} if lang_id_config is None else copy.deepcopy(lang_id_config)
+        self.lang_configs = {} if lang_configs is None else copy.deepcopy(lang_configs)
+        self.max_cache_size = max_cache_size
+        # OrderedDict so we can use it as a LRU cache
+        # most recent Pipeline goes to the end, pop the oldest one
+        # when we run out of space
+        self.pipeline_cache = OrderedDict()
+        if processors is None:
+            self.default_processors = None
+        elif isinstance(processors, str):
+            self.default_processors = [x.strip() for x in processors.split(",")]
+        else:
+            self.default_processors = list(processors)
+
+        self.download_method = download_method
+        if 'download_method' not in self.lang_id_config:
+            self.lang_id_config['download_method'] = self.download_method
+
+        # if lang is not in any of the lang_configs, update them to
+        # include the lang parameter.  otherwise, the default language
+        # will always be used...
+        for lang in self.lang_configs:
+            if 'lang' not in self.lang_configs[lang]:
+                self.lang_configs[lang]['lang'] = lang
+
+        if restrict and 'langid_lang_subset' not in self.lang_id_config:
+            known_langs = sorted(self.lang_configs.keys())
+            if known_langs == 0:
+                logger.warning("MultilingualPipeline asked to restrict to lang_configs, but lang_configs was empty.  Ignoring...")
+            else:
+                logger.debug("Restricting MultilingualPipeline to %s", known_langs)
+                self.lang_id_config['langid_lang_subset'] = known_langs
+
+        # set use_gpu
+        if device is None:
+            if use_gpu is None or use_gpu == True:
+                device = default_device()
+            else:
+                device = 'cpu'
+        self.device = device
+        
+        # build language id pipeline
+        self.lang_id_pipeline = Pipeline(dir=self.model_dir, lang='multilingual', processors="langid", 
+                                         device=self.device, **self.lang_id_config)
+        # load the resources so that we can refer to it later when building a new pipeline
+        # note that it was either downloaded or not based on download_method when building the lang_id_pipeline
+        self.resources = load_resources_json(self.model_dir)
+
+    def _update_pipeline_cache(self, lang):
+        """
+        Do any necessary updates to the pipeline cache for this language. This includes building a new
+        pipeline for the lang, and possibly clearing out a language with the old last access date.
+        """
+
+        # update request history
+        if lang in self.pipeline_cache:
+            self.pipeline_cache.move_to_end(lang, last=True)
+
+        # update language configs
+        # try/except to allow for a defaultdict
+        try:
+            lang_config = self.lang_configs[lang]
+        except KeyError:
+            lang_config = {'lang': lang}
+            self.lang_configs[lang] = lang_config
+
+        # if a defaultdict is passed in, the defaultdict might not contain 'lang'
+        # so even though we tried adding 'lang' in the constructor, we'll check again here
+        if 'lang' not in lang_config:
+            lang_config['lang'] = lang
+
+        if 'download_method' not in lang_config:
+            lang_config['download_method'] = self.download_method
+
+        if 'processors' not in lang_config:
+            if self.default_processors:
+                lang_resources = get_language_resources(self.resources, lang)
+                lang_processors = [x for x in self.default_processors if x in lang_resources]
+                if lang_processors != self.default_processors:
+                    logger.info("Not all requested processors %s available for %s.  Loading %s instead", self.default_processors, lang, lang_processors)
+                lang_config['processors'] = ",".join(lang_processors)
+
+        if 'device' not in lang_config:
+            lang_config['device'] = self.device
+
+        # update pipeline cache
+        if lang not in self.pipeline_cache:
+            logger.debug("Loading unknown language in MultilingualPipeline: %s", lang)
+            # clear least recently used lang from pipeline cache
+            if len(self.pipeline_cache) == self.max_cache_size:
+                self.pipeline_cache.popitem(last=False)
+            self.pipeline_cache[lang] = Pipeline(dir=self.model_dir, **self.lang_configs[lang])
+
+    def process(self, doc):
+        """
+        Run language detection on a string, a Document, or a list of either, route to language specific pipeline
+        """
+
+        # only return a list if given a list
+        singleton_input = not isinstance(doc, list)
+        if singleton_input:
+            docs = [doc]
+        else:
+            docs = doc
+
+        if docs and isinstance(docs[0], str):
+            docs = [Document([], text=text) for text in docs]
+
+        # run language identification
+        docs_w_langid = self.lang_id_pipeline.process(docs)
+
+        # create language specific batches, store global idx with each doc
+        lang_batches = {}
+        for doc_idx, doc in enumerate(docs_w_langid):
+            logger.debug("Language for document %d: %s", doc_idx, doc.lang)
+            if doc.lang not in lang_batches:
+                lang_batches[doc.lang] = []
+            lang_batches[doc.lang].append(doc)
+
+        # run through each language, submit a batch to the language specific pipeline
+        for lang in lang_batches.keys():
+            self._update_pipeline_cache(lang)
+            self.pipeline_cache[lang](lang_batches[lang])
+
+        # only return a list if given a list
+        if singleton_input:
+            return docs_w_langid[0]
+        else:
+            return docs_w_langid
+
+    def __call__(self, doc):
+        doc = self.process(doc)
+        return doc
+

stanza/stanza/pipeline/mwt_processor.py

@@ -0,0 +1,59 @@
+"""
+Processor for performing multi-word-token expansion
+"""
+
+import io
+
+import torch
+
+from stanza.models.mwt.data import DataLoader
+from stanza.models.mwt.trainer import Trainer
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+
+@register_processor(MWT)
+class MWTProcessor(UDProcessor):
+
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([MWT])
+    # set of processor requirements for this processor
+    REQUIRES_DEFAULT = set([TOKENIZE])
+
+    def _set_up_model(self, config, pipeline, device):
+        self._trainer = Trainer(model_file=config['model_path'], device=device)
+
+    def build_batch(self, document):
+        return DataLoader(document, self.config['batch_size'], self.config, vocab=self.vocab, evaluation=True, expand_unk_vocab=True)
+
+    def process(self, document):
+        batch = self.build_batch(document)
+
+        # process the rest
+        expansions = batch.doc.get_mwt_expansions(evaluation=True)
+        if len(batch) > 0:
+            # decide trainer type and run eval
+            if self.config['dict_only']:
+                preds = self.trainer.predict_dict(expansions)
+            else:
+                with torch.no_grad():
+                    preds = []
+                    for i, b in enumerate(batch.to_loader()):
+                        preds += self.trainer.predict(b, never_decode_unk=True, vocab=batch.vocab)
+
+                if self.config.get('ensemble_dict', False):
+                    preds = self.trainer.ensemble(expansions, preds)
+        else:
+            # skip eval if dev data does not exist
+            preds = []
+
+        batch.doc.set_mwt_expansions(preds, process_manual_expanded=False)
+        return batch.doc
+
+    def bulk_process(self, docs):
+        """
+        MWT processor counts some statistics on the individual docs, so we need to separately redo those stats
+        """
+        docs = super().bulk_process(docs)
+        for doc in docs:
+            doc._count_words()
+        return docs

stanza/stanza/pipeline/pos_processor.py

@@ -0,0 +1,89 @@
+"""
+Processor for performing part-of-speech tagging
+"""
+
+import torch
+
+from stanza.models.common import doc
+from stanza.models.common.utils import unsort
+from stanza.models.common.vocab import VOCAB_PREFIX, CompositeVocab
+from stanza.models.pos.data import Dataset
+from stanza.models.pos.trainer import Trainer
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+from stanza.utils.get_tqdm import get_tqdm
+
+tqdm = get_tqdm()
+
+@register_processor(name=POS)
+class POSProcessor(UDProcessor):
+
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([POS])
+    # set of processor requirements for this processor
+    REQUIRES_DEFAULT = set([TOKENIZE])
+
+    def _set_up_model(self, config, pipeline, device):
+        # get pretrained word vectors
+        self._pretrain = pipeline.foundation_cache.load_pretrain(config['pretrain_path']) if 'pretrain_path' in config else None
+        args = {'charlm_forward_file': config.get('forward_charlm_path', None),
+                'charlm_backward_file': config.get('backward_charlm_path', None)}
+        # set up trainer
+        self._trainer = Trainer(pretrain=self.pretrain, model_file=config['model_path'], device=device, args=args, foundation_cache=pipeline.foundation_cache)
+        self._tqdm = 'tqdm' in config and config['tqdm']
+
+    def __str__(self):
+        return "POSProcessor(%s)" % self.config['model_path']
+
+    def get_known_xpos(self):
+        """
+        Returns the xpos tags known by this model
+        """
+        if isinstance(self.vocab['xpos'], CompositeVocab):
+            if len(self.vocab['xpos']) == 1:
+                return [k for k in self.vocab['xpos'][0]._unit2id.keys() if k not in VOCAB_PREFIX]
+            else:
+                return {k: v.keys() - VOCAB_PREFIX for k, v in self.vocab['xpos']._unit2id.items()}
+        return [k for k in self.vocab['xpos']._unit2id.keys() if k not in VOCAB_PREFIX]
+
+    def is_composite_xpos(self):
+        """
+        Returns if the xpos tags are part of a composite vocab
+        """
+        return isinstance(self.vocab['xpos'], CompositeVocab)
+
+    def get_known_upos(self):
+        """
+        Returns the upos tags known by this model
+        """
+        keys = [k for k in self.vocab['upos']._unit2id.keys() if k not in VOCAB_PREFIX]
+        return keys
+
+    def get_known_feats(self):
+        """
+        Returns the features known by this model
+        """
+        values = {k: v.keys() - VOCAB_PREFIX for k, v in self.vocab['feats']._unit2id.items()}
+        return values
+
+    def process(self, document):
+        # currently, POS models are saved w/o the batch_maximum_tokens flag
+        maximum_tokens = self.config.get('batch_maximum_tokens', 5000)
+
+        dataset = Dataset(
+            document, self.config, self.pretrain, vocab=self.vocab, evaluation=True,
+            sort_during_eval=True)
+        batch = iter(dataset.to_length_limited_loader(batch_size=self.config['batch_size'], maximum_tokens=maximum_tokens))
+        preds = []
+
+        idx = []
+        with torch.no_grad():
+            if self._tqdm:
+                batch = tqdm(batch)
+            for i, b in enumerate(batch):
+                idx.extend(b[-1])
+                preds += self.trainer.predict(b)
+
+        preds = unsort(preds, idx)
+        dataset.doc.set([doc.UPOS, doc.XPOS, doc.FEATS], [y for x in preds for y in x])
+        return dataset.doc

stanza/stanza/pipeline/processor.py

@@ -0,0 +1,293 @@
+"""
+Base classes for processors
+"""
+
+from abc import ABC, abstractmethod
+
+from stanza.models.common.doc import Document
+from stanza.pipeline.registry import NAME_TO_PROCESSOR_CLASS, PIPELINE_NAMES, PROCESSOR_VARIANTS
+
+class ProcessorRequirementsException(Exception):
+    """ Exception indicating a processor's requirements will not be met """
+
+    def __init__(self, processors_list, err_processor, provided_reqs):
+        self._err_processor = err_processor
+        # mark the broken processor as inactive, drop resources
+        self.err_processor.mark_inactive()
+        self._processors_list = processors_list
+        self._provided_reqs = provided_reqs
+        self.build_message()
+
+    @property
+    def err_processor(self):
+        """ The processor that raised the exception """
+        return self._err_processor
+
+    @property
+    def processor_type(self):
+        return type(self.err_processor).__name__
+
+    @property
+    def processors_list(self):
+        return self._processors_list
+
+    @property
+    def provided_reqs(self):
+        return self._provided_reqs
+
+    def build_message(self):
+        self.message = (f"---\nPipeline Requirements Error!\n"
+                        f"\tProcessor: {self.processor_type}\n"
+                        f"\tPipeline processors list: {','.join(self.processors_list)}\n"
+                        f"\tProcessor Requirements: {self.err_processor.requires}\n"
+                        f"\t\t- fulfilled: {self.err_processor.requires.intersection(self.provided_reqs)}\n"
+                        f"\t\t- missing: {self.err_processor.requires - self.provided_reqs}\n"
+                        f"\nThe processors list provided for this pipeline is invalid.  Please make sure all "
+                        f"prerequisites are met for every processor.\n\n")
+
+    def __str__(self):
+        return self.message
+
+
+class Processor(ABC):
+    """ Base class for all processors """
+
+    def __init__(self, config, pipeline, device):
+        # overall config for the processor
+        self._config = config
+        # pipeline building this processor (presently processors are only meant to exist in one pipeline)
+        self._pipeline = pipeline
+        self._set_up_variants(config, device)
+        # run set up process
+        # set up what annotations are required based on config
+        if not self._set_up_variant_requires():
+            self._set_up_requires()
+        # set up what annotations are provided based on config
+        self._set_up_provides()
+        # given pipeline constructing this processor, check if requirements are met, throw exception if not
+        self._check_requirements()
+
+        if hasattr(self, '_variant') and self._variant.OVERRIDE:
+            self.process = self._variant.process
+
+    def __str__(self):
+        """
+        Simple description of the processor: name(model)
+        """
+        name = self.__class__.__name__
+        model = None
+        if self._config is not None:
+            model = self._config.get('model_path')
+        if model is None:
+            return name
+        else:
+            return "{}({})".format(name, model)
+
+
+    @abstractmethod
+    def process(self, doc):
+        """ Process a Document.  This is the main method of a processor. """
+        pass
+
+    def bulk_process(self, docs):
+        """ Process a list of Documents. This should be replaced with a more efficient implementation if possible. """
+
+        if hasattr(self, '_variant'):
+            return self._variant.bulk_process(docs)
+
+        return [self.process(doc) for doc in docs]
+
+    def _set_up_provides(self):
+        """ Set up what processor requirements this processor fulfills.  Default is to use a class defined list. """
+        self._provides = self.__class__.PROVIDES_DEFAULT
+
+    def _set_up_requires(self):
+        """ Set up requirements for this processor.  Default is to use a class defined list. """
+        self._requires = self.__class__.REQUIRES_DEFAULT
+
+    def _set_up_variant_requires(self):
+        """
+        If this has a variant with its own requirements, use those instead
+
+        Returns True iff the _requires is set from the _variant
+        """
+        if not hasattr(self, '_variant'):
+            return False
+        if hasattr(self._variant, '_set_up_requires'):
+            self._variant._set_up_requires()
+            self._requires = self._variant._requires
+            return True
+        if hasattr(self._variant.__class__, 'REQUIRES_DEFAULT'):
+            self._requires = self._variant.__class__.REQUIRES_DEFAULT
+            return True
+        return False
+
+    def _set_up_variants(self, config, device):
+        processor_name = list(self.__class__.PROVIDES_DEFAULT)[0]
+        if any(config.get(f'with_{variant}', False) for variant in PROCESSOR_VARIANTS[processor_name]):
+            self._trainer = None
+            variant_name = [variant for variant in PROCESSOR_VARIANTS[processor_name] if config.get(f'with_{variant}', False)][0]
+            self._variant = PROCESSOR_VARIANTS[processor_name][variant_name](config)
+
+    @property
+    def config(self):
+        """ Configurations for the processor """
+        return self._config
+
+    @property
+    def pipeline(self):
+        """ The pipeline that this processor belongs to """
+        return self._pipeline
+
+    @property
+    def provides(self):
+        return self._provides
+
+    @property
+    def requires(self):
+        return self._requires
+
+    def _check_requirements(self):
+        """ Given a list of fulfilled requirements, check if all of this processor's requirements are met or not. """
+        if not self.config.get("check_requirements", True):
+            return
+        provided_reqs = set.union(*[processor.provides for processor in self.pipeline.loaded_processors]+[set([])])
+        if self.requires - provided_reqs:
+            load_names = [item[0] for item in self.pipeline.load_list]
+            raise ProcessorRequirementsException(load_names, self, provided_reqs)
+
+
+class ProcessorVariant(ABC):
+    """ Base class for all processor variants """
+
+    OVERRIDE = False # Set to true to override all the processing from the processor
+
+    @abstractmethod
+    def process(self, doc):
+        """
+        Process a document that is potentially preprocessed by the processor.
+        This is the main method of a processor variant.
+
+        If `OVERRIDE` is set to True, all preprocessing by the processor would be bypassed, and the processor variant
+        would serve as a drop-in replacement of the entire processor, and has to be able to interpret all the configs
+        that are typically handled by the processor it replaces.
+        """
+        pass
+
+    def bulk_process(self, docs):
+        """ Process a list of Documents. This should be replaced with a more efficient implementation if possible. """
+
+        return [self.process(doc) for doc in docs]
+
+class UDProcessor(Processor):
+    """ Base class for the neural UD Processors (tokenize,mwt,pos,lemma,depparse,sentiment,constituency) """
+
+    def __init__(self, config, pipeline, device):
+        super().__init__(config, pipeline, device)
+
+        # UD model resources, set up is processor specific
+        self._pretrain = None
+        self._trainer = None
+        self._vocab = None
+        if not hasattr(self, '_variant'):
+            self._set_up_model(config, pipeline, device)
+
+        # build the final config for the processor
+        self._set_up_final_config(config)
+
+    @abstractmethod
+    def _set_up_model(self, config, pipeline, device):
+        pass
+
+    def _set_up_final_config(self, config):
+        """ Finalize the configurations for this processor, based off of values from a UD model. """
+        # set configurations from loaded model
+        if self._trainer is not None:
+            loaded_args, self._vocab = self._trainer.args, self._trainer.vocab
+            # filter out unneeded args from model
+            loaded_args = {k: v for k, v in loaded_args.items() if not UDProcessor.filter_out_option(k)}
+        else:
+            loaded_args = {}
+        loaded_args.update(config)
+        self._config = loaded_args
+
+    def mark_inactive(self):
+        """ Drop memory intensive resources if keeping this processor around for reasons other than running it. """
+        self._trainer = None
+        self._vocab = None
+
+    @property
+    def pretrain(self):
+        return self._pretrain
+
+    @property
+    def trainer(self):
+        return self._trainer
+
+    @property
+    def vocab(self):
+        return self._vocab
+
+    @staticmethod
+    def filter_out_option(option):
+        """ Filter out non-processor configurations """
+        options_to_filter = ['device', 'cpu', 'cuda', 'dev_conll_gold', 'epochs', 'lang', 'mode', 'save_name', 'shorthand']
+        if option.endswith('_file') or option.endswith('_dir'):
+            return True
+        elif option in options_to_filter:
+            return True
+        else:
+            return False
+
+    def bulk_process(self, docs):
+        """
+        Most processors operate on the sentence level, where each sentence is processed independently and processors can benefit
+        a lot from the ability to combine sentences from multiple documents for faster batched processing. This is a transparent
+        implementation that allows these processors to batch process a list of Documents as if they were from a single Document.
+        """
+
+        if hasattr(self, '_variant'):
+            return self._variant.bulk_process(docs)
+
+        combined_sents = [sent for doc in docs for sent in doc.sentences]
+        combined_doc = Document([])
+        combined_doc.sentences = combined_sents
+        combined_doc.num_tokens = sum(doc.num_tokens for doc in docs)
+        combined_doc.num_words = sum(doc.num_words for doc in docs)
+
+        self.process(combined_doc) # annotations are attached to sentence objects
+
+        return docs
+
+class ProcessorRegisterException(Exception):
+    """ Exception indicating processor or processor registration failure """
+
+    def __init__(self, processor_class, expected_parent):
+        self._processor_class = processor_class
+        self._expected_parent = expected_parent
+        self.build_message()
+
+    def build_message(self):
+        self.message = f"Failed to register '{self._processor_class}'. It must be a subclass of '{self._expected_parent}'."
+
+    def __str__(self):
+        return self.message
+
+def register_processor(name):
+    def wrapper(Cls):
+        if not issubclass(Cls, Processor):
+            raise ProcessorRegisterException(Cls, Processor)
+
+        NAME_TO_PROCESSOR_CLASS[name] = Cls
+        PIPELINE_NAMES.append(name)
+        return Cls
+    return wrapper
+
+def register_processor_variant(name, variant):
+    def wrapper(Cls):
+        if not issubclass(Cls, ProcessorVariant):
+            raise ProcessorRegisterException(Cls, ProcessorVariant)
+
+        PROCESSOR_VARIANTS[name][variant] = Cls
+        return Cls
+    return wrapper

stanza/stanza/pipeline/registry.py

@@ -0,0 +1,8 @@
+from collections import defaultdict
+
+# these two get filled by register_processor
+NAME_TO_PROCESSOR_CLASS = dict()
+PIPELINE_NAMES = []
+
+# this gets filled by register_processor_variant
+PROCESSOR_VARIANTS = defaultdict(dict)

stanza/stanza/pipeline/sentiment_processor.py

@@ -0,0 +1,78 @@
+"""Processor that attaches a sentiment score to a sentence
+
+The model used is a generally a model trained on the Stanford
+Sentiment Treebank or some similar dataset.  When run, this processor
+attaches a score in the form of a string to each sentence in the
+document.
+
+TODO: a possible way to generalize this would be to make it a
+ClassifierProcessor and have "sentiment" be an option.
+"""
+
+import dataclasses
+import torch
+
+from types import SimpleNamespace
+
+from stanza.models.classifiers.trainer import Trainer
+
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+
+@register_processor(SENTIMENT)
+class SentimentProcessor(UDProcessor):
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([SENTIMENT])
+    # set of processor requirements for this processor
+    # TODO: a constituency based model needs CONSTITUENCY as well
+    # issue: by the time we load the model in Processor.__init__,
+    # the requirements are already prepared
+    REQUIRES_DEFAULT = set([TOKENIZE])
+
+    # default batch size, measured in words per batch
+    DEFAULT_BATCH_SIZE = 5000
+
+    def _set_up_model(self, config, pipeline, device):
+        # get pretrained word vectors
+        pretrain_path = config.get('pretrain_path', None)
+        forward_charlm_path = config.get('forward_charlm_path', None)
+        backward_charlm_path = config.get('backward_charlm_path', None)
+        # elmo does not have a convenient way to download intermediate
+        # models the way stanza downloads charlms & pretrains or
+        # transformers downloads bert etc
+        # however, elmo in general is not as good as using a
+        # transformer, so it is unlikely we will ever fix this
+        args = SimpleNamespace(device = device,
+                               charlm_forward_file = forward_charlm_path,
+                               charlm_backward_file = backward_charlm_path,
+                               wordvec_pretrain_file = pretrain_path,
+                               elmo_model = None,
+                               use_elmo = False,
+                               save_dir = None)
+        filename = config['model_path']
+        if filename is None:
+            raise FileNotFoundError("No model specified for the sentiment processor.  Perhaps it is not supported for the language.  {}".format(config))
+        # set up model
+        trainer = Trainer.load(filename=filename,
+                               args=args,
+                               foundation_cache=pipeline.foundation_cache)
+        self._trainer = trainer
+        self._model = trainer.model
+        self._model_type = self._model.config.model_type
+        # batch size counted as words
+        self._batch_size = config.get('batch_size', SentimentProcessor.DEFAULT_BATCH_SIZE)
+
+    def _set_up_final_config(self, config):
+        loaded_args = dataclasses.asdict(self._model.config)
+        loaded_args = {k: v for k, v in loaded_args.items() if not UDProcessor.filter_out_option(k)}
+        loaded_args.update(config)
+        self._config = loaded_args
+
+
+    def process(self, document):
+        sentences = self._model.extract_sentences(document)
+        with torch.no_grad():
+            labels = self._model.label_sentences(sentences, batch_size=self._batch_size)
+        # TODO: allow a classifier processor for any attribute, not just sentiment
+        document.set(SENTIMENT, labels, to_sentence=True)
+        return document

stanza/stanza/pipeline/tokenize_processor.py

@@ -0,0 +1,185 @@
+"""
+Processor for performing tokenization
+"""
+
+import io
+import logging
+
+import torch
+
+from stanza.models.tokenization.data import TokenizationDataset
+from stanza.models.tokenization.trainer import Trainer
+from stanza.models.tokenization.utils import output_predictions
+from stanza.pipeline._constants import *
+from stanza.pipeline.processor import UDProcessor, register_processor
+from stanza.pipeline.registry import PROCESSOR_VARIANTS
+from stanza.models.common import doc
+
+# these imports trigger the "register_variant" decorations
+from stanza.pipeline.external.jieba import JiebaTokenizer
+from stanza.pipeline.external.spacy import SpacyTokenizer
+from stanza.pipeline.external.sudachipy import SudachiPyTokenizer
+from stanza.pipeline.external.pythainlp import PyThaiNLPTokenizer
+
+logger = logging.getLogger('stanza')
+
+TOKEN_TOO_LONG_REPLACEMENT = "<UNK>"
+
+# class for running the tokenizer
+@register_processor(name=TOKENIZE)
+class TokenizeProcessor(UDProcessor):
+
+    # set of processor requirements this processor fulfills
+    PROVIDES_DEFAULT = set([TOKENIZE])
+    # set of processor requirements for this processor
+    REQUIRES_DEFAULT = set([])
+    # default max sequence length
+    MAX_SEQ_LENGTH_DEFAULT = 1000
+
+    def _set_up_model(self, config, pipeline, device):
+        # set up trainer
+        if config.get('pretokenized'):
+            self._trainer = None
+        else:
+            self._trainer = Trainer(model_file=config['model_path'], device=device)
+
+        # get and typecheck the postprocessor
+        postprocessor = config.get('postprocessor')
+        if postprocessor and callable(postprocessor):
+            self._postprocessor = postprocessor
+        elif not postprocessor:
+            self._postprocessor = None
+        else:
+            raise ValueError("Tokenizer recieved 'postprocessor' option of unrecognized type; postprocessor must be callable. Got %s" % postprocessor)
+
+    def process_pre_tokenized_text(self, input_src):
+        """
+        Pretokenized text can be provided in 2 manners:
+
+        1.) str, tokenized by whitespace, sentence split by newline
+        2.) list of token lists, each token list represents a sentence
+
+        generate dictionary data structure
+        """
+
+        document = []
+        if isinstance(input_src, str):
+            sentences = [sent.strip().split() for sent in input_src.strip().split('\n') if len(sent.strip()) > 0]
+        elif isinstance(input_src, list):
+            sentences = input_src
+        idx = 0
+        for sentence in sentences:
+            sent = []
+            for token_id, token in enumerate(sentence):
+                sent.append({doc.ID: (token_id + 1, ), doc.TEXT: token, doc.MISC: f'start_char={idx}|end_char={idx + len(token)}'})
+                idx += len(token) + 1
+            document.append(sent)
+        raw_text = ' '.join([' '.join(sentence) for sentence in sentences])
+        return raw_text, document
+
+    def process(self, document):
+        if not (isinstance(document, str) or isinstance(document, doc.Document) or (self.config.get('pretokenized') or self.config.get('no_ssplit', False))):
+            raise ValueError("If neither 'pretokenized' or 'no_ssplit' option is enabled, the input to the TokenizerProcessor must be a string or a Document object.  Got %s" % str(type(document)))
+
+        if isinstance(document, doc.Document):
+            if self.config.get('pretokenized'):
+                return document
+            document = document.text
+
+        if self.config.get('pretokenized'):
+            raw_text, document = self.process_pre_tokenized_text(document)
+            return doc.Document(document, raw_text)
+
+        if hasattr(self, '_variant'):
+            return self._variant.process(document)
+
+        raw_text = '\n\n'.join(document) if isinstance(document, list) else document
+
+        max_seq_len = self.config.get('max_seqlen', TokenizeProcessor.MAX_SEQ_LENGTH_DEFAULT)
+
+        # set up batches
+        batches = TokenizationDataset(self.config, input_text=raw_text, vocab=self.vocab, evaluation=True, dictionary=self.trainer.dictionary)
+        # get dict data
+        with torch.no_grad():
+            _, _, _, document = output_predictions(None, self.trainer, batches, self.vocab, None,
+                                                   max_seq_len,
+                                                   orig_text=raw_text,
+                                                   no_ssplit=self.config.get('no_ssplit', False),
+                                                   num_workers = self.config.get('num_workers', 0),
+                                                   postprocessor = self._postprocessor)
+
+        # replace excessively long tokens with <UNK> to avoid downstream GPU memory issues in POS
+        for sentence in document:
+            for token in sentence:
+                if len(token['text']) > max_seq_len:
+                    token['text'] = TOKEN_TOO_LONG_REPLACEMENT
+
+        return doc.Document(document, raw_text)
+
+    def bulk_process(self, docs):
+        """
+        The tokenizer cannot use UDProcessor's sentence-level cross-document batching interface, and requires special handling.
+        Essentially, this method concatenates the text of multiple documents with "\n\n", tokenizes it with the neural tokenizer,
+        then splits the result into the original Documents and recovers the original character offsets.
+        """
+        if hasattr(self, '_variant'):
+            return self._variant.bulk_process(docs)
+
+        if self.config.get('pretokenized'):
+            res = []
+            for document in docs:
+                raw_text, document = self.process_pre_tokenized_text(document.text)
+                res.append(doc.Document(document, raw_text))
+            return res
+
+        combined_text = '\n\n'.join([thisdoc.text for thisdoc in docs])
+        processed_combined = self.process(doc.Document([], text=combined_text))
+
+        # postprocess sentences and tokens to reset back pointers and char offsets
+        charoffset = 0
+        sentst = senten = 0
+        for thisdoc in docs:
+            while senten < len(processed_combined.sentences) and processed_combined.sentences[senten].tokens[-1].end_char - charoffset <= len(thisdoc.text):
+                senten += 1
+
+            sentences = processed_combined.sentences[sentst:senten]
+            thisdoc.sentences = sentences
+            for sent in sentences:
+                # fix doc back pointers for sentences
+                sent._doc = thisdoc
+
+                # fix char offsets for tokens and words
+                for token in sent.tokens:
+                    token._start_char -= charoffset
+                    token._end_char -= charoffset
+                    if token.words:  # not-yet-processed MWT can leave empty tokens
+                        for word in token.words:
+                            word._start_char -= charoffset
+                            word._end_char -= charoffset
+
+            # Here we need to fix up the SpacesAfter for the very last token
+            # and the SpacesBefore for the first token of the next doc
+            # After all, we had connected the text with \n\n
+            # Need to be careful about this - in a case such as
+            #   " -text one- "
+            #   " -text two- "
+            # We want the SpacesBefore for the second document to reflect
+            # the extra space at the start of its text
+            # and the SpacesAfter for the first document to reflect
+            # the whitespace after its text
+            if len(sentences) > 0:
+                last_token = sentences[-1].tokens[-1]
+                last_whitespace = thisdoc.text[last_token.end_char:]
+                last_token.spaces_after = last_whitespace
+
+                first_token = sentences[0].tokens[0]
+                first_whitespace = thisdoc.text[:first_token.start_char]
+                first_token.spaces_before = first_whitespace
+
+            thisdoc.num_tokens = sum(len(sent.tokens) for sent in sentences)
+            thisdoc.num_words = sum(len(sent.words) for sent in sentences)
+            sentst = senten
+
+            charoffset += len(thisdoc.text) + 2
+
+        return docs

stanza/stanza/protobuf/CoreNLP_pb2.py

@@ -0,0 +1,686 @@
+# -*- coding: utf-8 -*-
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# source: CoreNLP.proto
+"""Generated protocol buffer code."""
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import descriptor_pool as _descriptor_pool
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+
+
+DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\rCoreNLP.proto\x12\x19\x65\x64u.stanford.nlp.pipeline\"\xe1\x05\n\x08\x44ocument\x12\x0c\n\x04text\x18\x01 \x02(\t\x12\x35\n\x08sentence\x18\x02 \x03(\x0b\x32#.edu.stanford.nlp.pipeline.Sentence\x12\x39\n\ncorefChain\x18\x03 \x03(\x0b\x32%.edu.stanford.nlp.pipeline.CorefChain\x12\r\n\x05\x64ocID\x18\x04 \x01(\t\x12\x0f\n\x07\x64ocDate\x18\x07 \x01(\t\x12\x10\n\x08\x63\x61lendar\x18\x08 \x01(\x04\x12;\n\x11sentencelessToken\x18\x05 \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Token\x12\x33\n\tcharacter\x18\n \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Token\x12/\n\x05quote\x18\x06 \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Quote\x12\x37\n\x08mentions\x18\t \x03(\x0b\x32%.edu.stanford.nlp.pipeline.NERMention\x12#\n\x1bhasEntityMentionsAnnotation\x18\r \x01(\x08\x12\x0e\n\x06xmlDoc\x18\x0b \x01(\x08\x12\x34\n\x08sections\x18\x0c \x03(\x0b\x32\".edu.stanford.nlp.pipeline.Section\x12<\n\x10mentionsForCoref\x18\x0e \x03(\x0b\x32\".edu.stanford.nlp.pipeline.Mention\x12!\n\x19hasCorefMentionAnnotation\x18\x0f \x01(\x08\x12\x1a\n\x12hasCorefAnnotation\x18\x10 \x01(\x08\x12+\n#corefMentionToEntityMentionMappings\x18\x11 \x03(\x05\x12+\n#entityMentionToCorefMentionMappings\x18\x12 \x03(\x05*\x05\x08\x64\x10\x80\x02\"\xf3\x0f\n\x08Sentence\x12/\n\x05token\x18\x01 \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Token\x12\x18\n\x10tokenOffsetBegin\x18\x02 \x02(\r\x12\x16\n\x0etokenOffsetEnd\x18\x03 \x02(\r\x12\x15\n\rsentenceIndex\x18\x04 \x01(\r\x12\x1c\n\x14\x63haracterOffsetBegin\x18\x05 \x01(\r\x12\x1a\n\x12\x63haracterOffsetEnd\x18\x06 \x01(\r\x12\x37\n\tparseTree\x18\x07 \x01(\x0b\x32$.edu.stanford.nlp.pipeline.ParseTree\x12@\n\x12\x62inarizedParseTree\x18\x1f \x01(\x0b\x32$.edu.stanford.nlp.pipeline.ParseTree\x12@\n\x12\x61nnotatedParseTree\x18  \x01(\x0b\x32$.edu.stanford.nlp.pipeline.ParseTree\x12\x11\n\tsentiment\x18! \x01(\t\x12=\n\x0fkBestParseTrees\x18\" \x03(\x0b\x32$.edu.stanford.nlp.pipeline.ParseTree\x12\x45\n\x11\x62\x61sicDependencies\x18\x08 \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12I\n\x15\x63ollapsedDependencies\x18\t \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12T\n collapsedCCProcessedDependencies\x18\n \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12K\n\x17\x61lternativeDependencies\x18\r \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12?\n\x0copenieTriple\x18\x0e \x03(\x0b\x32).edu.stanford.nlp.pipeline.RelationTriple\x12<\n\tkbpTriple\x18\x10 \x03(\x0b\x32).edu.stanford.nlp.pipeline.RelationTriple\x12\x45\n\x10\x65ntailedSentence\x18\x0f \x03(\x0b\x32+.edu.stanford.nlp.pipeline.SentenceFragment\x12\x43\n\x0e\x65ntailedClause\x18# \x03(\x0b\x32+.edu.stanford.nlp.pipeline.SentenceFragment\x12H\n\x14\x65nhancedDependencies\x18\x11 \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12P\n\x1c\x65nhancedPlusPlusDependencies\x18\x12 \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12\x33\n\tcharacter\x18\x13 \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Token\x12\x11\n\tparagraph\x18\x0b \x01(\r\x12\x0c\n\x04text\x18\x0c \x01(\t\x12\x12\n\nlineNumber\x18\x14 \x01(\r\x12\x1e\n\x16hasRelationAnnotations\x18\x33 \x01(\x08\x12\x31\n\x06\x65ntity\x18\x34 \x03(\x0b\x32!.edu.stanford.nlp.pipeline.Entity\x12\x35\n\x08relation\x18\x35 \x03(\x0b\x32#.edu.stanford.nlp.pipeline.Relation\x12$\n\x1chasNumerizedTokensAnnotation\x18\x36 \x01(\x08\x12\x37\n\x08mentions\x18\x37 \x03(\x0b\x32%.edu.stanford.nlp.pipeline.NERMention\x12<\n\x10mentionsForCoref\x18\x38 \x03(\x0b\x32\".edu.stanford.nlp.pipeline.Mention\x12\"\n\x1ahasCorefMentionsAnnotation\x18\x39 \x01(\x08\x12\x12\n\nsentenceID\x18: \x01(\t\x12\x13\n\x0bsectionDate\x18; \x01(\t\x12\x14\n\x0csectionIndex\x18< \x01(\r\x12\x13\n\x0bsectionName\x18= \x01(\t\x12\x15\n\rsectionAuthor\x18> \x01(\t\x12\r\n\x05\x64ocID\x18? \x01(\t\x12\x15\n\rsectionQuoted\x18@ \x01(\x08\x12#\n\x1bhasEntityMentionsAnnotation\x18\x41 \x01(\x08\x12\x1f\n\x17hasKBPTriplesAnnotation\x18\x44 \x01(\x08\x12\"\n\x1ahasOpenieTriplesAnnotation\x18\x45 \x01(\x08\x12\x14\n\x0c\x63hapterIndex\x18\x42 \x01(\r\x12\x16\n\x0eparagraphIndex\x18\x43 \x01(\r\x12=\n\x10\x65nhancedSentence\x18\x46 \x01(\x0b\x32#.edu.stanford.nlp.pipeline.Sentence\x12\x0f\n\x07speaker\x18G \x01(\t\x12\x13\n\x0bspeakerType\x18H \x01(\t*\x05\x08\x64\x10\x80\x02\"\xf6\x0c\n\x05Token\x12\x0c\n\x04word\x18\x01 \x01(\t\x12\x0b\n\x03pos\x18\x02 \x01(\t\x12\r\n\x05value\x18\x03 \x01(\t\x12\x10\n\x08\x63\x61tegory\x18\x04 \x01(\t\x12\x0e\n\x06\x62\x65\x66ore\x18\x05 \x01(\t\x12\r\n\x05\x61\x66ter\x18\x06 \x01(\t\x12\x14\n\x0coriginalText\x18\x07 \x01(\t\x12\x0b\n\x03ner\x18\x08 \x01(\t\x12\x11\n\tcoarseNER\x18> \x01(\t\x12\x16\n\x0e\x66ineGrainedNER\x18? \x01(\t\x12\x15\n\rnerLabelProbs\x18\x42 \x03(\t\x12\x15\n\rnormalizedNER\x18\t \x01(\t\x12\r\n\x05lemma\x18\n \x01(\t\x12\x11\n\tbeginChar\x18\x0b \x01(\r\x12\x0f\n\x07\x65ndChar\x18\x0c \x01(\r\x12\x11\n\tutterance\x18\r \x01(\r\x12\x0f\n\x07speaker\x18\x0e \x01(\t\x12\x13\n\x0bspeakerType\x18M \x01(\t\x12\x12\n\nbeginIndex\x18\x0f \x01(\r\x12\x10\n\x08\x65ndIndex\x18\x10 \x01(\r\x12\x17\n\x0ftokenBeginIndex\x18\x11 \x01(\r\x12\x15\n\rtokenEndIndex\x18\x12 \x01(\r\x12\x34\n\ntimexValue\x18\x13 \x01(\x0b\x32 .edu.stanford.nlp.pipeline.Timex\x12\x15\n\rhasXmlContext\x18\x15 \x01(\x08\x12\x12\n\nxmlContext\x18\x16 \x03(\t\x12\x16\n\x0e\x63orefClusterID\x18\x17 \x01(\r\x12\x0e\n\x06\x61nswer\x18\x18 \x01(\t\x12\x15\n\rheadWordIndex\x18\x1a \x01(\r\x12\x35\n\x08operator\x18\x1b \x01(\x0b\x32#.edu.stanford.nlp.pipeline.Operator\x12\x35\n\x08polarity\x18\x1c \x01(\x0b\x32#.edu.stanford.nlp.pipeline.Polarity\x12\x14\n\x0cpolarity_dir\x18\' \x01(\t\x12-\n\x04span\x18\x1d \x01(\x0b\x32\x1f.edu.stanford.nlp.pipeline.Span\x12\x11\n\tsentiment\x18\x1e \x01(\t\x12\x16\n\x0equotationIndex\x18\x1f \x01(\x05\x12\x42\n\x0e\x63onllUFeatures\x18  \x01(\x0b\x32*.edu.stanford.nlp.pipeline.MapStringString\x12\x11\n\tcoarseTag\x18! \x01(\t\x12\x38\n\x0f\x63onllUTokenSpan\x18\" \x01(\x0b\x32\x1f.edu.stanford.nlp.pipeline.Span\x12\x12\n\nconllUMisc\x18# \x01(\t\x12G\n\x13\x63onllUSecondaryDeps\x18$ \x01(\x0b\x32*.edu.stanford.nlp.pipeline.MapStringString\x12\x17\n\x0fwikipediaEntity\x18% \x01(\t\x12\x11\n\tisNewline\x18& \x01(\x08\x12\x0e\n\x06gender\x18\x33 \x01(\t\x12\x10\n\x08trueCase\x18\x34 \x01(\t\x12\x14\n\x0ctrueCaseText\x18\x35 \x01(\t\x12\x13\n\x0b\x63hineseChar\x18\x36 \x01(\t\x12\x12\n\nchineseSeg\x18\x37 \x01(\t\x12\x16\n\x0e\x63hineseXMLChar\x18< \x01(\t\x12\x11\n\tarabicSeg\x18L \x01(\t\x12\x13\n\x0bsectionName\x18\x38 \x01(\t\x12\x15\n\rsectionAuthor\x18\x39 \x01(\t\x12\x13\n\x0bsectionDate\x18: \x01(\t\x12\x17\n\x0fsectionEndLabel\x18; \x01(\t\x12\x0e\n\x06parent\x18= \x01(\t\x12\x19\n\x11\x63orefMentionIndex\x18@ \x03(\r\x12\x1a\n\x12\x65ntityMentionIndex\x18\x41 \x01(\r\x12\r\n\x05isMWT\x18\x43 \x01(\x08\x12\x12\n\nisFirstMWT\x18\x44 \x01(\x08\x12\x0f\n\x07mwtText\x18\x45 \x01(\t\x12\x0f\n\x07mwtMisc\x18N \x01(\t\x12\x14\n\x0cnumericValue\x18\x46 \x01(\x04\x12\x13\n\x0bnumericType\x18G \x01(\t\x12\x1d\n\x15numericCompositeValue\x18H \x01(\x04\x12\x1c\n\x14numericCompositeType\x18I \x01(\t\x12\x1c\n\x14\x63odepointOffsetBegin\x18J \x01(\r\x12\x1a\n\x12\x63odepointOffsetEnd\x18K \x01(\r\x12\r\n\x05index\x18O \x01(\r\x12\x12\n\nemptyIndex\x18P \x01(\r*\x05\x08\x64\x10\x80\x02\"\xe4\x03\n\x05Quote\x12\x0c\n\x04text\x18\x01 \x01(\t\x12\r\n\x05\x62\x65gin\x18\x02 \x01(\r\x12\x0b\n\x03\x65nd\x18\x03 \x01(\r\x12\x15\n\rsentenceBegin\x18\x05 \x01(\r\x12\x13\n\x0bsentenceEnd\x18\x06 \x01(\r\x12\x12\n\ntokenBegin\x18\x07 \x01(\r\x12\x10\n\x08tokenEnd\x18\x08 \x01(\r\x12\r\n\x05\x64ocid\x18\t \x01(\t\x12\r\n\x05index\x18\n \x01(\r\x12\x0e\n\x06\x61uthor\x18\x0b \x01(\t\x12\x0f\n\x07mention\x18\x0c \x01(\t\x12\x14\n\x0cmentionBegin\x18\r \x01(\r\x12\x12\n\nmentionEnd\x18\x0e \x01(\r\x12\x13\n\x0bmentionType\x18\x0f \x01(\t\x12\x14\n\x0cmentionSieve\x18\x10 \x01(\t\x12\x0f\n\x07speaker\x18\x11 \x01(\t\x12\x14\n\x0cspeakerSieve\x18\x12 \x01(\t\x12\x18\n\x10\x63\x61nonicalMention\x18\x13 \x01(\t\x12\x1d\n\x15\x63\x61nonicalMentionBegin\x18\x14 \x01(\r\x12\x1b\n\x13\x63\x61nonicalMentionEnd\x18\x15 \x01(\r\x12N\n\x1a\x61ttributionDependencyGraph\x18\x16 \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\"\xc7\x01\n\tParseTree\x12\x33\n\x05\x63hild\x18\x01 \x03(\x0b\x32$.edu.stanford.nlp.pipeline.ParseTree\x12\r\n\x05value\x18\x02 \x01(\t\x12\x17\n\x0fyieldBeginIndex\x18\x03 \x01(\r\x12\x15\n\ryieldEndIndex\x18\x04 \x01(\r\x12\r\n\x05score\x18\x05 \x01(\x01\x12\x37\n\tsentiment\x18\x06 \x01(\x0e\x32$.edu.stanford.nlp.pipeline.Sentiment\"\x9b\x04\n\x0f\x44\x65pendencyGraph\x12=\n\x04node\x18\x01 \x03(\x0b\x32/.edu.stanford.nlp.pipeline.DependencyGraph.Node\x12=\n\x04\x65\x64ge\x18\x02 \x03(\x0b\x32/.edu.stanford.nlp.pipeline.DependencyGraph.Edge\x12\x10\n\x04root\x18\x03 \x03(\rB\x02\x10\x01\x12/\n\x05token\x18\x04 \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Token\x12\x14\n\x08rootNode\x18\x05 \x03(\rB\x02\x10\x01\x1aX\n\x04Node\x12\x15\n\rsentenceIndex\x18\x01 \x02(\r\x12\r\n\x05index\x18\x02 \x02(\r\x12\x16\n\x0e\x63opyAnnotation\x18\x03 \x01(\r\x12\x12\n\nemptyIndex\x18\x04 \x01(\r\x1a\xd6\x01\n\x04\x45\x64ge\x12\x0e\n\x06source\x18\x01 \x02(\r\x12\x0e\n\x06target\x18\x02 \x02(\r\x12\x0b\n\x03\x64\x65p\x18\x03 \x01(\t\x12\x0f\n\x07isExtra\x18\x04 \x01(\x08\x12\x12\n\nsourceCopy\x18\x05 \x01(\r\x12\x12\n\ntargetCopy\x18\x06 \x01(\r\x12\x13\n\x0bsourceEmpty\x18\x08 \x01(\r\x12\x13\n\x0btargetEmpty\x18\t \x01(\r\x12>\n\x08language\x18\x07 \x01(\x0e\x32#.edu.stanford.nlp.pipeline.Language:\x07Unknown\"\xc6\x02\n\nCorefChain\x12\x0f\n\x07\x63hainID\x18\x01 \x02(\x05\x12\x43\n\x07mention\x18\x02 \x03(\x0b\x32\x32.edu.stanford.nlp.pipeline.CorefChain.CorefMention\x12\x16\n\x0erepresentative\x18\x03 \x02(\r\x1a\xc9\x01\n\x0c\x43orefMention\x12\x11\n\tmentionID\x18\x01 \x01(\x05\x12\x13\n\x0bmentionType\x18\x02 \x01(\t\x12\x0e\n\x06number\x18\x03 \x01(\t\x12\x0e\n\x06gender\x18\x04 \x01(\t\x12\x0f\n\x07\x61nimacy\x18\x05 \x01(\t\x12\x12\n\nbeginIndex\x18\x06 \x01(\r\x12\x10\n\x08\x65ndIndex\x18\x07 \x01(\r\x12\x11\n\theadIndex\x18\t \x01(\r\x12\x15\n\rsentenceIndex\x18\n \x01(\r\x12\x10\n\x08position\x18\x0b \x01(\r\"\xef\x08\n\x07Mention\x12\x11\n\tmentionID\x18\x01 \x01(\x05\x12\x13\n\x0bmentionType\x18\x02 \x01(\t\x12\x0e\n\x06number\x18\x03 \x01(\t\x12\x0e\n\x06gender\x18\x04 \x01(\t\x12\x0f\n\x07\x61nimacy\x18\x05 \x01(\t\x12\x0e\n\x06person\x18\x06 \x01(\t\x12\x12\n\nstartIndex\x18\x07 \x01(\r\x12\x10\n\x08\x65ndIndex\x18\t \x01(\r\x12\x11\n\theadIndex\x18\n \x01(\x05\x12\x12\n\nheadString\x18\x0b \x01(\t\x12\x11\n\tnerString\x18\x0c \x01(\t\x12\x13\n\x0boriginalRef\x18\r \x01(\x05\x12\x1a\n\x12goldCorefClusterID\x18\x0e \x01(\x05\x12\x16\n\x0e\x63orefClusterID\x18\x0f \x01(\x05\x12\x12\n\nmentionNum\x18\x10 \x01(\x05\x12\x0f\n\x07sentNum\x18\x11 \x01(\x05\x12\r\n\x05utter\x18\x12 \x01(\x05\x12\x11\n\tparagraph\x18\x13 \x01(\x05\x12\x11\n\tisSubject\x18\x14 \x01(\x08\x12\x16\n\x0eisDirectObject\x18\x15 \x01(\x08\x12\x18\n\x10isIndirectObject\x18\x16 \x01(\x08\x12\x1b\n\x13isPrepositionObject\x18\x17 \x01(\x08\x12\x0f\n\x07hasTwin\x18\x18 \x01(\x08\x12\x0f\n\x07generic\x18\x19 \x01(\x08\x12\x13\n\x0bisSingleton\x18\x1a \x01(\x08\x12\x1a\n\x12hasBasicDependency\x18\x1b \x01(\x08\x12\x1d\n\x15hasEnhancedDependency\x18\x1c \x01(\x08\x12\x1b\n\x13hasContextParseTree\x18\x1d \x01(\x08\x12?\n\x0fheadIndexedWord\x18\x1e \x01(\x0b\x32&.edu.stanford.nlp.pipeline.IndexedWord\x12=\n\rdependingVerb\x18\x1f \x01(\x0b\x32&.edu.stanford.nlp.pipeline.IndexedWord\x12\x38\n\x08headWord\x18  \x01(\x0b\x32&.edu.stanford.nlp.pipeline.IndexedWord\x12;\n\x0bspeakerInfo\x18! \x01(\x0b\x32&.edu.stanford.nlp.pipeline.SpeakerInfo\x12=\n\rsentenceWords\x18\x32 \x03(\x0b\x32&.edu.stanford.nlp.pipeline.IndexedWord\x12<\n\x0coriginalSpan\x18\x33 \x03(\x0b\x32&.edu.stanford.nlp.pipeline.IndexedWord\x12\x12\n\ndependents\x18\x34 \x03(\t\x12\x19\n\x11preprocessedTerms\x18\x35 \x03(\t\x12\x13\n\x0b\x61ppositions\x18\x36 \x03(\x05\x12\x1c\n\x14predicateNominatives\x18\x37 \x03(\x05\x12\x18\n\x10relativePronouns\x18\x38 \x03(\x05\x12\x13\n\x0blistMembers\x18\x39 \x03(\x05\x12\x15\n\rbelongToLists\x18: \x03(\x05\"X\n\x0bIndexedWord\x12\x13\n\x0bsentenceNum\x18\x01 \x01(\x05\x12\x12\n\ntokenIndex\x18\x02 \x01(\x05\x12\r\n\x05\x64ocID\x18\x03 \x01(\x05\x12\x11\n\tcopyCount\x18\x04 \x01(\r\"4\n\x0bSpeakerInfo\x12\x13\n\x0bspeakerName\x18\x01 \x01(\t\x12\x10\n\x08mentions\x18\x02 \x03(\x05\"\"\n\x04Span\x12\r\n\x05\x62\x65gin\x18\x01 \x02(\r\x12\x0b\n\x03\x65nd\x18\x02 \x02(\r\"w\n\x05Timex\x12\r\n\x05value\x18\x01 \x01(\t\x12\x10\n\x08\x61ltValue\x18\x02 \x01(\t\x12\x0c\n\x04text\x18\x03 \x01(\t\x12\x0c\n\x04type\x18\x04 \x01(\t\x12\x0b\n\x03tid\x18\x05 \x01(\t\x12\x12\n\nbeginPoint\x18\x06 \x01(\r\x12\x10\n\x08\x65ndPoint\x18\x07 \x01(\r\"\xdb\x01\n\x06\x45ntity\x12\x11\n\theadStart\x18\x06 \x01(\r\x12\x0f\n\x07headEnd\x18\x07 \x01(\r\x12\x13\n\x0bmentionType\x18\x08 \x01(\t\x12\x16\n\x0enormalizedName\x18\t \x01(\t\x12\x16\n\x0eheadTokenIndex\x18\n \x01(\r\x12\x0f\n\x07\x63orefID\x18\x0b \x01(\t\x12\x10\n\x08objectID\x18\x01 \x01(\t\x12\x13\n\x0b\x65xtentStart\x18\x02 \x01(\r\x12\x11\n\textentEnd\x18\x03 \x01(\r\x12\x0c\n\x04type\x18\x04 \x01(\t\x12\x0f\n\x07subtype\x18\x05 \x01(\t\"\xb7\x01\n\x08Relation\x12\x0f\n\x07\x61rgName\x18\x06 \x03(\t\x12.\n\x03\x61rg\x18\x07 \x03(\x0b\x32!.edu.stanford.nlp.pipeline.Entity\x12\x11\n\tsignature\x18\x08 \x01(\t\x12\x10\n\x08objectID\x18\x01 \x01(\t\x12\x13\n\x0b\x65xtentStart\x18\x02 \x01(\r\x12\x11\n\textentEnd\x18\x03 \x01(\r\x12\x0c\n\x04type\x18\x04 \x01(\t\x12\x0f\n\x07subtype\x18\x05 \x01(\t\"\xb2\x01\n\x08Operator\x12\x0c\n\x04name\x18\x01 \x02(\t\x12\x1b\n\x13quantifierSpanBegin\x18\x02 \x02(\x05\x12\x19\n\x11quantifierSpanEnd\x18\x03 \x02(\x05\x12\x18\n\x10subjectSpanBegin\x18\x04 \x02(\x05\x12\x16\n\x0esubjectSpanEnd\x18\x05 \x02(\x05\x12\x17\n\x0fobjectSpanBegin\x18\x06 \x02(\x05\x12\x15\n\robjectSpanEnd\x18\x07 \x02(\x05\"\xa9\x04\n\x08Polarity\x12K\n\x12projectEquivalence\x18\x01 \x02(\x0e\x32/.edu.stanford.nlp.pipeline.NaturalLogicRelation\x12Q\n\x18projectForwardEntailment\x18\x02 \x02(\x0e\x32/.edu.stanford.nlp.pipeline.NaturalLogicRelation\x12Q\n\x18projectReverseEntailment\x18\x03 \x02(\x0e\x32/.edu.stanford.nlp.pipeline.NaturalLogicRelation\x12H\n\x0fprojectNegation\x18\x04 \x02(\x0e\x32/.edu.stanford.nlp.pipeline.NaturalLogicRelation\x12K\n\x12projectAlternation\x18\x05 \x02(\x0e\x32/.edu.stanford.nlp.pipeline.NaturalLogicRelation\x12\x45\n\x0cprojectCover\x18\x06 \x02(\x0e\x32/.edu.stanford.nlp.pipeline.NaturalLogicRelation\x12L\n\x13projectIndependence\x18\x07 \x02(\x0e\x32/.edu.stanford.nlp.pipeline.NaturalLogicRelation\"\xdd\x02\n\nNERMention\x12\x15\n\rsentenceIndex\x18\x01 \x01(\r\x12%\n\x1dtokenStartInSentenceInclusive\x18\x02 \x02(\r\x12#\n\x1btokenEndInSentenceExclusive\x18\x03 \x02(\r\x12\x0b\n\x03ner\x18\x04 \x02(\t\x12\x15\n\rnormalizedNER\x18\x05 \x01(\t\x12\x12\n\nentityType\x18\x06 \x01(\t\x12/\n\x05timex\x18\x07 \x01(\x0b\x32 .edu.stanford.nlp.pipeline.Timex\x12\x17\n\x0fwikipediaEntity\x18\x08 \x01(\t\x12\x0e\n\x06gender\x18\t \x01(\t\x12\x1a\n\x12\x65ntityMentionIndex\x18\n \x01(\r\x12#\n\x1b\x63\x61nonicalEntityMentionIndex\x18\x0b \x01(\r\x12\x19\n\x11\x65ntityMentionText\x18\x0c \x01(\t\"Y\n\x10SentenceFragment\x12\x12\n\ntokenIndex\x18\x01 \x03(\r\x12\x0c\n\x04root\x18\x02 \x01(\r\x12\x14\n\x0c\x61ssumedTruth\x18\x03 \x01(\x08\x12\r\n\x05score\x18\x04 \x01(\x01\":\n\rTokenLocation\x12\x15\n\rsentenceIndex\x18\x01 \x01(\r\x12\x12\n\ntokenIndex\x18\x02 \x01(\r\"\x9a\x03\n\x0eRelationTriple\x12\x0f\n\x07subject\x18\x01 \x01(\t\x12\x10\n\x08relation\x18\x02 \x01(\t\x12\x0e\n\x06object\x18\x03 \x01(\t\x12\x12\n\nconfidence\x18\x04 \x01(\x01\x12?\n\rsubjectTokens\x18\r \x03(\x0b\x32(.edu.stanford.nlp.pipeline.TokenLocation\x12@\n\x0erelationTokens\x18\x0e \x03(\x0b\x32(.edu.stanford.nlp.pipeline.TokenLocation\x12>\n\x0cobjectTokens\x18\x0f \x03(\x0b\x32(.edu.stanford.nlp.pipeline.TokenLocation\x12\x38\n\x04tree\x18\x08 \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12\x0e\n\x06istmod\x18\t \x01(\x08\x12\x10\n\x08prefixBe\x18\n \x01(\x08\x12\x10\n\x08suffixBe\x18\x0b \x01(\x08\x12\x10\n\x08suffixOf\x18\x0c \x01(\x08\"-\n\x0fMapStringString\x12\x0b\n\x03key\x18\x01 \x03(\t\x12\r\n\x05value\x18\x02 \x03(\t\"*\n\x0cMapIntString\x12\x0b\n\x03key\x18\x01 \x03(\r\x12\r\n\x05value\x18\x02 \x03(\t\"\xfc\x01\n\x07Section\x12\x11\n\tcharBegin\x18\x01 \x02(\r\x12\x0f\n\x07\x63harEnd\x18\x02 \x02(\r\x12\x0e\n\x06\x61uthor\x18\x03 \x01(\t\x12\x17\n\x0fsentenceIndexes\x18\x04 \x03(\r\x12\x10\n\x08\x64\x61tetime\x18\x05 \x01(\t\x12\x30\n\x06quotes\x18\x06 \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Quote\x12\x17\n\x0f\x61uthorCharBegin\x18\x07 \x01(\r\x12\x15\n\rauthorCharEnd\x18\x08 \x01(\r\x12\x30\n\x06xmlTag\x18\t \x02(\x0b\x32 .edu.stanford.nlp.pipeline.Token\"\xe4\x01\n\x0eSemgrexRequest\x12\x0f\n\x07semgrex\x18\x01 \x03(\t\x12\x45\n\x05query\x18\x02 \x03(\x0b\x32\x36.edu.stanford.nlp.pipeline.SemgrexRequest.Dependencies\x1az\n\x0c\x44\x65pendencies\x12/\n\x05token\x18\x01 \x03(\x0b\x32 .edu.stanford.nlp.pipeline.Token\x12\x39\n\x05graph\x18\x02 \x02(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\"\xfb\x05\n\x0fSemgrexResponse\x12\x46\n\x06result\x18\x01 \x03(\x0b\x32\x36.edu.stanford.nlp.pipeline.SemgrexResponse.GraphResult\x1a-\n\tNamedNode\x12\x0c\n\x04name\x18\x01 \x02(\t\x12\x12\n\nmatchIndex\x18\x02 \x02(\x05\x1a+\n\rNamedRelation\x12\x0c\n\x04name\x18\x01 \x02(\t\x12\x0c\n\x04reln\x18\x02 \x02(\t\x1a\x80\x01\n\tNamedEdge\x12\x0c\n\x04name\x18\x01 \x02(\t\x12\x0e\n\x06source\x18\x02 \x02(\x05\x12\x0e\n\x06target\x18\x03 \x02(\x05\x12\x0c\n\x04reln\x18\x04 \x01(\t\x12\x0f\n\x07isExtra\x18\x05 \x01(\x08\x12\x12\n\nsourceCopy\x18\x06 \x01(\r\x12\x12\n\ntargetCopy\x18\x07 \x01(\r\x1a\x95\x02\n\x05Match\x12\x12\n\nmatchIndex\x18\x01 \x02(\x05\x12\x42\n\x04node\x18\x02 \x03(\x0b\x32\x34.edu.stanford.nlp.pipeline.SemgrexResponse.NamedNode\x12\x46\n\x04reln\x18\x03 \x03(\x0b\x32\x38.edu.stanford.nlp.pipeline.SemgrexResponse.NamedRelation\x12\x42\n\x04\x65\x64ge\x18\x06 \x03(\x0b\x32\x34.edu.stanford.nlp.pipeline.SemgrexResponse.NamedEdge\x12\x12\n\ngraphIndex\x18\x04 \x01(\x05\x12\x14\n\x0csemgrexIndex\x18\x05 \x01(\x05\x1aP\n\rSemgrexResult\x12?\n\x05match\x18\x01 \x03(\x0b\x32\x30.edu.stanford.nlp.pipeline.SemgrexResponse.Match\x1aW\n\x0bGraphResult\x12H\n\x06result\x18\x01 \x03(\x0b\x32\x38.edu.stanford.nlp.pipeline.SemgrexResponse.SemgrexResult\"\xf0\x01\n\x0fSsurgeonRequest\x12\x45\n\x08ssurgeon\x18\x01 \x03(\x0b\x32\x33.edu.stanford.nlp.pipeline.SsurgeonRequest.Ssurgeon\x12\x39\n\x05graph\x18\x02 \x03(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x1a[\n\x08Ssurgeon\x12\x0f\n\x07semgrex\x18\x01 \x01(\t\x12\x11\n\toperation\x18\x02 \x03(\t\x12\n\n\x02id\x18\x03 \x01(\t\x12\r\n\x05notes\x18\x04 \x01(\t\x12\x10\n\x08language\x18\x05 \x01(\t\"\xbc\x01\n\x10SsurgeonResponse\x12J\n\x06result\x18\x01 \x03(\x0b\x32:.edu.stanford.nlp.pipeline.SsurgeonResponse.SsurgeonResult\x1a\\\n\x0eSsurgeonResult\x12\x39\n\x05graph\x18\x01 \x01(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12\x0f\n\x07\x63hanged\x18\x02 \x01(\x08\"W\n\x12TokensRegexRequest\x12\x30\n\x03\x64oc\x18\x01 \x02(\x0b\x32#.edu.stanford.nlp.pipeline.Document\x12\x0f\n\x07pattern\x18\x02 \x03(\t\"\xa7\x03\n\x13TokensRegexResponse\x12J\n\x05match\x18\x01 \x03(\x0b\x32;.edu.stanford.nlp.pipeline.TokensRegexResponse.PatternMatch\x1a\x39\n\rMatchLocation\x12\x0c\n\x04text\x18\x01 \x01(\t\x12\r\n\x05\x62\x65gin\x18\x02 \x01(\x05\x12\x0b\n\x03\x65nd\x18\x03 \x01(\x05\x1a\xb3\x01\n\x05Match\x12\x10\n\x08sentence\x18\x01 \x02(\x05\x12K\n\x05match\x18\x02 \x02(\x0b\x32<.edu.stanford.nlp.pipeline.TokensRegexResponse.MatchLocation\x12K\n\x05group\x18\x03 \x03(\x0b\x32<.edu.stanford.nlp.pipeline.TokensRegexResponse.MatchLocation\x1aS\n\x0cPatternMatch\x12\x43\n\x05match\x18\x01 \x03(\x0b\x32\x34.edu.stanford.nlp.pipeline.TokensRegexResponse.Match\"\xae\x01\n\x19\x44\x65pendencyEnhancerRequest\x12\x35\n\x08\x64ocument\x18\x01 \x02(\x0b\x32#.edu.stanford.nlp.pipeline.Document\x12\x37\n\x08language\x18\x02 \x01(\x0e\x32#.edu.stanford.nlp.pipeline.LanguageH\x00\x12\x1a\n\x10relativePronouns\x18\x03 \x01(\tH\x00\x42\x05\n\x03ref\"\xb4\x01\n\x12\x46lattenedParseTree\x12\x41\n\x05nodes\x18\x01 \x03(\x0b\x32\x32.edu.stanford.nlp.pipeline.FlattenedParseTree.Node\x1a[\n\x04Node\x12\x12\n\x08openNode\x18\x01 \x01(\x08H\x00\x12\x13\n\tcloseNode\x18\x02 \x01(\x08H\x00\x12\x0f\n\x05value\x18\x03 \x01(\tH\x00\x12\r\n\x05score\x18\x04 \x01(\x01\x42\n\n\x08\x63ontents\"\xf6\x01\n\x15\x45valuateParserRequest\x12N\n\x08treebank\x18\x01 \x03(\x0b\x32<.edu.stanford.nlp.pipeline.EvaluateParserRequest.ParseResult\x1a\x8c\x01\n\x0bParseResult\x12;\n\x04gold\x18\x01 \x02(\x0b\x32-.edu.stanford.nlp.pipeline.FlattenedParseTree\x12@\n\tpredicted\x18\x02 \x03(\x0b\x32-.edu.stanford.nlp.pipeline.FlattenedParseTree\"E\n\x16\x45valuateParserResponse\x12\n\n\x02\x66\x31\x18\x01 \x02(\x01\x12\x0f\n\x07kbestF1\x18\x02 \x01(\x01\x12\x0e\n\x06treeF1\x18\x03 \x03(\x01\"\xc8\x01\n\x0fTsurgeonRequest\x12H\n\noperations\x18\x01 \x03(\x0b\x32\x34.edu.stanford.nlp.pipeline.TsurgeonRequest.Operation\x12<\n\x05trees\x18\x02 \x03(\x0b\x32-.edu.stanford.nlp.pipeline.FlattenedParseTree\x1a-\n\tOperation\x12\x0e\n\x06tregex\x18\x01 \x02(\t\x12\x10\n\x08tsurgeon\x18\x02 \x03(\t\"P\n\x10TsurgeonResponse\x12<\n\x05trees\x18\x01 \x03(\x0b\x32-.edu.stanford.nlp.pipeline.FlattenedParseTree\"\x85\x01\n\x11MorphologyRequest\x12\x46\n\x05words\x18\x01 \x03(\x0b\x32\x37.edu.stanford.nlp.pipeline.MorphologyRequest.TaggedWord\x1a(\n\nTaggedWord\x12\x0c\n\x04word\x18\x01 \x02(\t\x12\x0c\n\x04xpos\x18\x02 \x01(\t\"\x9a\x01\n\x12MorphologyResponse\x12I\n\x05words\x18\x01 \x03(\x0b\x32:.edu.stanford.nlp.pipeline.MorphologyResponse.WordTagLemma\x1a\x39\n\x0cWordTagLemma\x12\x0c\n\x04word\x18\x01 \x02(\t\x12\x0c\n\x04xpos\x18\x02 \x01(\t\x12\r\n\x05lemma\x18\x03 \x02(\t\"Z\n\x1a\x44\x65pendencyConverterRequest\x12<\n\x05trees\x18\x01 \x03(\x0b\x32-.edu.stanford.nlp.pipeline.FlattenedParseTree\"\x90\x02\n\x1b\x44\x65pendencyConverterResponse\x12`\n\x0b\x63onversions\x18\x01 \x03(\x0b\x32K.edu.stanford.nlp.pipeline.DependencyConverterResponse.DependencyConversion\x1a\x8e\x01\n\x14\x44\x65pendencyConversion\x12\x39\n\x05graph\x18\x01 \x02(\x0b\x32*.edu.stanford.nlp.pipeline.DependencyGraph\x12;\n\x04tree\x18\x02 \x01(\x0b\x32-.edu.stanford.nlp.pipeline.FlattenedParseTree*\xa3\x01\n\x08Language\x12\x0b\n\x07Unknown\x10\x00\x12\x07\n\x03\x41ny\x10\x01\x12\n\n\x06\x41rabic\x10\x02\x12\x0b\n\x07\x43hinese\x10\x03\x12\x0b\n\x07\x45nglish\x10\x04\x12\n\n\x06German\x10\x05\x12\n\n\x06\x46rench\x10\x06\x12\n\n\x06Hebrew\x10\x07\x12\x0b\n\x07Spanish\x10\x08\x12\x14\n\x10UniversalEnglish\x10\t\x12\x14\n\x10UniversalChinese\x10\n*h\n\tSentiment\x12\x13\n\x0fSTRONG_NEGATIVE\x10\x00\x12\x11\n\rWEAK_NEGATIVE\x10\x01\x12\x0b\n\x07NEUTRAL\x10\x02\x12\x11\n\rWEAK_POSITIVE\x10\x03\x12\x13\n\x0fSTRONG_POSITIVE\x10\x04*\x93\x01\n\x14NaturalLogicRelation\x12\x0f\n\x0b\x45QUIVALENCE\x10\x00\x12\x16\n\x12\x46ORWARD_ENTAILMENT\x10\x01\x12\x16\n\x12REVERSE_ENTAILMENT\x10\x02\x12\x0c\n\x08NEGATION\x10\x03\x12\x0f\n\x0b\x41LTERNATION\x10\x04\x12\t\n\x05\x43OVER\x10\x05\x12\x10\n\x0cINDEPENDENCE\x10\x06\x42*\n\x19\x65\x64u.stanford.nlp.pipelineB\rCoreNLPProtos')
+
+_LANGUAGE = DESCRIPTOR.enum_types_by_name['Language']
+Language = enum_type_wrapper.EnumTypeWrapper(_LANGUAGE)
+_SENTIMENT = DESCRIPTOR.enum_types_by_name['Sentiment']
+Sentiment = enum_type_wrapper.EnumTypeWrapper(_SENTIMENT)
+_NATURALLOGICRELATION = DESCRIPTOR.enum_types_by_name['NaturalLogicRelation']
+NaturalLogicRelation = enum_type_wrapper.EnumTypeWrapper(_NATURALLOGICRELATION)
+Unknown = 0
+Any = 1
+Arabic = 2
+Chinese = 3
+English = 4
+German = 5
+French = 6
+Hebrew = 7
+Spanish = 8
+UniversalEnglish = 9
+UniversalChinese = 10
+STRONG_NEGATIVE = 0
+WEAK_NEGATIVE = 1
+NEUTRAL = 2
+WEAK_POSITIVE = 3
+STRONG_POSITIVE = 4
+EQUIVALENCE = 0
+FORWARD_ENTAILMENT = 1
+REVERSE_ENTAILMENT = 2
+NEGATION = 3
+ALTERNATION = 4
+COVER = 5
+INDEPENDENCE = 6
+
+
+_DOCUMENT = DESCRIPTOR.message_types_by_name['Document']
+_SENTENCE = DESCRIPTOR.message_types_by_name['Sentence']
+_TOKEN = DESCRIPTOR.message_types_by_name['Token']
+_QUOTE = DESCRIPTOR.message_types_by_name['Quote']
+_PARSETREE = DESCRIPTOR.message_types_by_name['ParseTree']
+_DEPENDENCYGRAPH = DESCRIPTOR.message_types_by_name['DependencyGraph']
+_DEPENDENCYGRAPH_NODE = _DEPENDENCYGRAPH.nested_types_by_name['Node']
+_DEPENDENCYGRAPH_EDGE = _DEPENDENCYGRAPH.nested_types_by_name['Edge']
+_COREFCHAIN = DESCRIPTOR.message_types_by_name['CorefChain']
+_COREFCHAIN_COREFMENTION = _COREFCHAIN.nested_types_by_name['CorefMention']
+_MENTION = DESCRIPTOR.message_types_by_name['Mention']
+_INDEXEDWORD = DESCRIPTOR.message_types_by_name['IndexedWord']
+_SPEAKERINFO = DESCRIPTOR.message_types_by_name['SpeakerInfo']
+_SPAN = DESCRIPTOR.message_types_by_name['Span']
+_TIMEX = DESCRIPTOR.message_types_by_name['Timex']
+_ENTITY = DESCRIPTOR.message_types_by_name['Entity']
+_RELATION = DESCRIPTOR.message_types_by_name['Relation']
+_OPERATOR = DESCRIPTOR.message_types_by_name['Operator']
+_POLARITY = DESCRIPTOR.message_types_by_name['Polarity']
+_NERMENTION = DESCRIPTOR.message_types_by_name['NERMention']
+_SENTENCEFRAGMENT = DESCRIPTOR.message_types_by_name['SentenceFragment']
+_TOKENLOCATION = DESCRIPTOR.message_types_by_name['TokenLocation']
+_RELATIONTRIPLE = DESCRIPTOR.message_types_by_name['RelationTriple']
+_MAPSTRINGSTRING = DESCRIPTOR.message_types_by_name['MapStringString']
+_MAPINTSTRING = DESCRIPTOR.message_types_by_name['MapIntString']
+_SECTION = DESCRIPTOR.message_types_by_name['Section']
+_SEMGREXREQUEST = DESCRIPTOR.message_types_by_name['SemgrexRequest']
+_SEMGREXREQUEST_DEPENDENCIES = _SEMGREXREQUEST.nested_types_by_name['Dependencies']
+_SEMGREXRESPONSE = DESCRIPTOR.message_types_by_name['SemgrexResponse']
+_SEMGREXRESPONSE_NAMEDNODE = _SEMGREXRESPONSE.nested_types_by_name['NamedNode']
+_SEMGREXRESPONSE_NAMEDRELATION = _SEMGREXRESPONSE.nested_types_by_name['NamedRelation']
+_SEMGREXRESPONSE_NAMEDEDGE = _SEMGREXRESPONSE.nested_types_by_name['NamedEdge']
+_SEMGREXRESPONSE_MATCH = _SEMGREXRESPONSE.nested_types_by_name['Match']
+_SEMGREXRESPONSE_SEMGREXRESULT = _SEMGREXRESPONSE.nested_types_by_name['SemgrexResult']
+_SEMGREXRESPONSE_GRAPHRESULT = _SEMGREXRESPONSE.nested_types_by_name['GraphResult']
+_SSURGEONREQUEST = DESCRIPTOR.message_types_by_name['SsurgeonRequest']
+_SSURGEONREQUEST_SSURGEON = _SSURGEONREQUEST.nested_types_by_name['Ssurgeon']
+_SSURGEONRESPONSE = DESCRIPTOR.message_types_by_name['SsurgeonResponse']
+_SSURGEONRESPONSE_SSURGEONRESULT = _SSURGEONRESPONSE.nested_types_by_name['SsurgeonResult']
+_TOKENSREGEXREQUEST = DESCRIPTOR.message_types_by_name['TokensRegexRequest']
+_TOKENSREGEXRESPONSE = DESCRIPTOR.message_types_by_name['TokensRegexResponse']
+_TOKENSREGEXRESPONSE_MATCHLOCATION = _TOKENSREGEXRESPONSE.nested_types_by_name['MatchLocation']
+_TOKENSREGEXRESPONSE_MATCH = _TOKENSREGEXRESPONSE.nested_types_by_name['Match']
+_TOKENSREGEXRESPONSE_PATTERNMATCH = _TOKENSREGEXRESPONSE.nested_types_by_name['PatternMatch']
+_DEPENDENCYENHANCERREQUEST = DESCRIPTOR.message_types_by_name['DependencyEnhancerRequest']
+_FLATTENEDPARSETREE = DESCRIPTOR.message_types_by_name['FlattenedParseTree']
+_FLATTENEDPARSETREE_NODE = _FLATTENEDPARSETREE.nested_types_by_name['Node']
+_EVALUATEPARSERREQUEST = DESCRIPTOR.message_types_by_name['EvaluateParserRequest']
+_EVALUATEPARSERREQUEST_PARSERESULT = _EVALUATEPARSERREQUEST.nested_types_by_name['ParseResult']
+_EVALUATEPARSERRESPONSE = DESCRIPTOR.message_types_by_name['EvaluateParserResponse']
+_TSURGEONREQUEST = DESCRIPTOR.message_types_by_name['TsurgeonRequest']
+_TSURGEONREQUEST_OPERATION = _TSURGEONREQUEST.nested_types_by_name['Operation']
+_TSURGEONRESPONSE = DESCRIPTOR.message_types_by_name['TsurgeonResponse']
+_MORPHOLOGYREQUEST = DESCRIPTOR.message_types_by_name['MorphologyRequest']
+_MORPHOLOGYREQUEST_TAGGEDWORD = _MORPHOLOGYREQUEST.nested_types_by_name['TaggedWord']
+_MORPHOLOGYRESPONSE = DESCRIPTOR.message_types_by_name['MorphologyResponse']
+_MORPHOLOGYRESPONSE_WORDTAGLEMMA = _MORPHOLOGYRESPONSE.nested_types_by_name['WordTagLemma']
+_DEPENDENCYCONVERTERREQUEST = DESCRIPTOR.message_types_by_name['DependencyConverterRequest']
+_DEPENDENCYCONVERTERRESPONSE = DESCRIPTOR.message_types_by_name['DependencyConverterResponse']
+_DEPENDENCYCONVERTERRESPONSE_DEPENDENCYCONVERSION = _DEPENDENCYCONVERTERRESPONSE.nested_types_by_name['DependencyConversion']
+Document = _reflection.GeneratedProtocolMessageType('Document', (_message.Message,), {
+  'DESCRIPTOR' : _DOCUMENT,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Document)
+  })
+_sym_db.RegisterMessage(Document)
+
+Sentence = _reflection.GeneratedProtocolMessageType('Sentence', (_message.Message,), {
+  'DESCRIPTOR' : _SENTENCE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Sentence)
+  })
+_sym_db.RegisterMessage(Sentence)
+
+Token = _reflection.GeneratedProtocolMessageType('Token', (_message.Message,), {
+  'DESCRIPTOR' : _TOKEN,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Token)
+  })
+_sym_db.RegisterMessage(Token)
+
+Quote = _reflection.GeneratedProtocolMessageType('Quote', (_message.Message,), {
+  'DESCRIPTOR' : _QUOTE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Quote)
+  })
+_sym_db.RegisterMessage(Quote)
+
+ParseTree = _reflection.GeneratedProtocolMessageType('ParseTree', (_message.Message,), {
+  'DESCRIPTOR' : _PARSETREE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.ParseTree)
+  })
+_sym_db.RegisterMessage(ParseTree)
+
+DependencyGraph = _reflection.GeneratedProtocolMessageType('DependencyGraph', (_message.Message,), {
+
+  'Node' : _reflection.GeneratedProtocolMessageType('Node', (_message.Message,), {
+    'DESCRIPTOR' : _DEPENDENCYGRAPH_NODE,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.DependencyGraph.Node)
+    })
+  ,
+
+  'Edge' : _reflection.GeneratedProtocolMessageType('Edge', (_message.Message,), {
+    'DESCRIPTOR' : _DEPENDENCYGRAPH_EDGE,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.DependencyGraph.Edge)
+    })
+  ,
+  'DESCRIPTOR' : _DEPENDENCYGRAPH,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.DependencyGraph)
+  })
+_sym_db.RegisterMessage(DependencyGraph)
+_sym_db.RegisterMessage(DependencyGraph.Node)
+_sym_db.RegisterMessage(DependencyGraph.Edge)
+
+CorefChain = _reflection.GeneratedProtocolMessageType('CorefChain', (_message.Message,), {
+
+  'CorefMention' : _reflection.GeneratedProtocolMessageType('CorefMention', (_message.Message,), {
+    'DESCRIPTOR' : _COREFCHAIN_COREFMENTION,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.CorefChain.CorefMention)
+    })
+  ,
+  'DESCRIPTOR' : _COREFCHAIN,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.CorefChain)
+  })
+_sym_db.RegisterMessage(CorefChain)
+_sym_db.RegisterMessage(CorefChain.CorefMention)
+
+Mention = _reflection.GeneratedProtocolMessageType('Mention', (_message.Message,), {
+  'DESCRIPTOR' : _MENTION,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Mention)
+  })
+_sym_db.RegisterMessage(Mention)
+
+IndexedWord = _reflection.GeneratedProtocolMessageType('IndexedWord', (_message.Message,), {
+  'DESCRIPTOR' : _INDEXEDWORD,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.IndexedWord)
+  })
+_sym_db.RegisterMessage(IndexedWord)
+
+SpeakerInfo = _reflection.GeneratedProtocolMessageType('SpeakerInfo', (_message.Message,), {
+  'DESCRIPTOR' : _SPEAKERINFO,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SpeakerInfo)
+  })
+_sym_db.RegisterMessage(SpeakerInfo)
+
+Span = _reflection.GeneratedProtocolMessageType('Span', (_message.Message,), {
+  'DESCRIPTOR' : _SPAN,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Span)
+  })
+_sym_db.RegisterMessage(Span)
+
+Timex = _reflection.GeneratedProtocolMessageType('Timex', (_message.Message,), {
+  'DESCRIPTOR' : _TIMEX,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Timex)
+  })
+_sym_db.RegisterMessage(Timex)
+
+Entity = _reflection.GeneratedProtocolMessageType('Entity', (_message.Message,), {
+  'DESCRIPTOR' : _ENTITY,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Entity)
+  })
+_sym_db.RegisterMessage(Entity)
+
+Relation = _reflection.GeneratedProtocolMessageType('Relation', (_message.Message,), {
+  'DESCRIPTOR' : _RELATION,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Relation)
+  })
+_sym_db.RegisterMessage(Relation)
+
+Operator = _reflection.GeneratedProtocolMessageType('Operator', (_message.Message,), {
+  'DESCRIPTOR' : _OPERATOR,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Operator)
+  })
+_sym_db.RegisterMessage(Operator)
+
+Polarity = _reflection.GeneratedProtocolMessageType('Polarity', (_message.Message,), {
+  'DESCRIPTOR' : _POLARITY,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Polarity)
+  })
+_sym_db.RegisterMessage(Polarity)
+
+NERMention = _reflection.GeneratedProtocolMessageType('NERMention', (_message.Message,), {
+  'DESCRIPTOR' : _NERMENTION,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.NERMention)
+  })
+_sym_db.RegisterMessage(NERMention)
+
+SentenceFragment = _reflection.GeneratedProtocolMessageType('SentenceFragment', (_message.Message,), {
+  'DESCRIPTOR' : _SENTENCEFRAGMENT,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SentenceFragment)
+  })
+_sym_db.RegisterMessage(SentenceFragment)
+
+TokenLocation = _reflection.GeneratedProtocolMessageType('TokenLocation', (_message.Message,), {
+  'DESCRIPTOR' : _TOKENLOCATION,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TokenLocation)
+  })
+_sym_db.RegisterMessage(TokenLocation)
+
+RelationTriple = _reflection.GeneratedProtocolMessageType('RelationTriple', (_message.Message,), {
+  'DESCRIPTOR' : _RELATIONTRIPLE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.RelationTriple)
+  })
+_sym_db.RegisterMessage(RelationTriple)
+
+MapStringString = _reflection.GeneratedProtocolMessageType('MapStringString', (_message.Message,), {
+  'DESCRIPTOR' : _MAPSTRINGSTRING,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.MapStringString)
+  })
+_sym_db.RegisterMessage(MapStringString)
+
+MapIntString = _reflection.GeneratedProtocolMessageType('MapIntString', (_message.Message,), {
+  'DESCRIPTOR' : _MAPINTSTRING,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.MapIntString)
+  })
+_sym_db.RegisterMessage(MapIntString)
+
+Section = _reflection.GeneratedProtocolMessageType('Section', (_message.Message,), {
+  'DESCRIPTOR' : _SECTION,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.Section)
+  })
+_sym_db.RegisterMessage(Section)
+
+SemgrexRequest = _reflection.GeneratedProtocolMessageType('SemgrexRequest', (_message.Message,), {
+
+  'Dependencies' : _reflection.GeneratedProtocolMessageType('Dependencies', (_message.Message,), {
+    'DESCRIPTOR' : _SEMGREXREQUEST_DEPENDENCIES,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexRequest.Dependencies)
+    })
+  ,
+  'DESCRIPTOR' : _SEMGREXREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexRequest)
+  })
+_sym_db.RegisterMessage(SemgrexRequest)
+_sym_db.RegisterMessage(SemgrexRequest.Dependencies)
+
+SemgrexResponse = _reflection.GeneratedProtocolMessageType('SemgrexResponse', (_message.Message,), {
+
+  'NamedNode' : _reflection.GeneratedProtocolMessageType('NamedNode', (_message.Message,), {
+    'DESCRIPTOR' : _SEMGREXRESPONSE_NAMEDNODE,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexResponse.NamedNode)
+    })
+  ,
+
+  'NamedRelation' : _reflection.GeneratedProtocolMessageType('NamedRelation', (_message.Message,), {
+    'DESCRIPTOR' : _SEMGREXRESPONSE_NAMEDRELATION,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexResponse.NamedRelation)
+    })
+  ,
+
+  'NamedEdge' : _reflection.GeneratedProtocolMessageType('NamedEdge', (_message.Message,), {
+    'DESCRIPTOR' : _SEMGREXRESPONSE_NAMEDEDGE,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexResponse.NamedEdge)
+    })
+  ,
+
+  'Match' : _reflection.GeneratedProtocolMessageType('Match', (_message.Message,), {
+    'DESCRIPTOR' : _SEMGREXRESPONSE_MATCH,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexResponse.Match)
+    })
+  ,
+
+  'SemgrexResult' : _reflection.GeneratedProtocolMessageType('SemgrexResult', (_message.Message,), {
+    'DESCRIPTOR' : _SEMGREXRESPONSE_SEMGREXRESULT,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexResponse.SemgrexResult)
+    })
+  ,
+
+  'GraphResult' : _reflection.GeneratedProtocolMessageType('GraphResult', (_message.Message,), {
+    'DESCRIPTOR' : _SEMGREXRESPONSE_GRAPHRESULT,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexResponse.GraphResult)
+    })
+  ,
+  'DESCRIPTOR' : _SEMGREXRESPONSE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SemgrexResponse)
+  })
+_sym_db.RegisterMessage(SemgrexResponse)
+_sym_db.RegisterMessage(SemgrexResponse.NamedNode)
+_sym_db.RegisterMessage(SemgrexResponse.NamedRelation)
+_sym_db.RegisterMessage(SemgrexResponse.NamedEdge)
+_sym_db.RegisterMessage(SemgrexResponse.Match)
+_sym_db.RegisterMessage(SemgrexResponse.SemgrexResult)
+_sym_db.RegisterMessage(SemgrexResponse.GraphResult)
+
+SsurgeonRequest = _reflection.GeneratedProtocolMessageType('SsurgeonRequest', (_message.Message,), {
+
+  'Ssurgeon' : _reflection.GeneratedProtocolMessageType('Ssurgeon', (_message.Message,), {
+    'DESCRIPTOR' : _SSURGEONREQUEST_SSURGEON,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SsurgeonRequest.Ssurgeon)
+    })
+  ,
+  'DESCRIPTOR' : _SSURGEONREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SsurgeonRequest)
+  })
+_sym_db.RegisterMessage(SsurgeonRequest)
+_sym_db.RegisterMessage(SsurgeonRequest.Ssurgeon)
+
+SsurgeonResponse = _reflection.GeneratedProtocolMessageType('SsurgeonResponse', (_message.Message,), {
+
+  'SsurgeonResult' : _reflection.GeneratedProtocolMessageType('SsurgeonResult', (_message.Message,), {
+    'DESCRIPTOR' : _SSURGEONRESPONSE_SSURGEONRESULT,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SsurgeonResponse.SsurgeonResult)
+    })
+  ,
+  'DESCRIPTOR' : _SSURGEONRESPONSE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.SsurgeonResponse)
+  })
+_sym_db.RegisterMessage(SsurgeonResponse)
+_sym_db.RegisterMessage(SsurgeonResponse.SsurgeonResult)
+
+TokensRegexRequest = _reflection.GeneratedProtocolMessageType('TokensRegexRequest', (_message.Message,), {
+  'DESCRIPTOR' : _TOKENSREGEXREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TokensRegexRequest)
+  })
+_sym_db.RegisterMessage(TokensRegexRequest)
+
+TokensRegexResponse = _reflection.GeneratedProtocolMessageType('TokensRegexResponse', (_message.Message,), {
+
+  'MatchLocation' : _reflection.GeneratedProtocolMessageType('MatchLocation', (_message.Message,), {
+    'DESCRIPTOR' : _TOKENSREGEXRESPONSE_MATCHLOCATION,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TokensRegexResponse.MatchLocation)
+    })
+  ,
+
+  'Match' : _reflection.GeneratedProtocolMessageType('Match', (_message.Message,), {
+    'DESCRIPTOR' : _TOKENSREGEXRESPONSE_MATCH,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TokensRegexResponse.Match)
+    })
+  ,
+
+  'PatternMatch' : _reflection.GeneratedProtocolMessageType('PatternMatch', (_message.Message,), {
+    'DESCRIPTOR' : _TOKENSREGEXRESPONSE_PATTERNMATCH,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TokensRegexResponse.PatternMatch)
+    })
+  ,
+  'DESCRIPTOR' : _TOKENSREGEXRESPONSE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TokensRegexResponse)
+  })
+_sym_db.RegisterMessage(TokensRegexResponse)
+_sym_db.RegisterMessage(TokensRegexResponse.MatchLocation)
+_sym_db.RegisterMessage(TokensRegexResponse.Match)
+_sym_db.RegisterMessage(TokensRegexResponse.PatternMatch)
+
+DependencyEnhancerRequest = _reflection.GeneratedProtocolMessageType('DependencyEnhancerRequest', (_message.Message,), {
+  'DESCRIPTOR' : _DEPENDENCYENHANCERREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.DependencyEnhancerRequest)
+  })
+_sym_db.RegisterMessage(DependencyEnhancerRequest)
+
+FlattenedParseTree = _reflection.GeneratedProtocolMessageType('FlattenedParseTree', (_message.Message,), {
+
+  'Node' : _reflection.GeneratedProtocolMessageType('Node', (_message.Message,), {
+    'DESCRIPTOR' : _FLATTENEDPARSETREE_NODE,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.FlattenedParseTree.Node)
+    })
+  ,
+  'DESCRIPTOR' : _FLATTENEDPARSETREE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.FlattenedParseTree)
+  })
+_sym_db.RegisterMessage(FlattenedParseTree)
+_sym_db.RegisterMessage(FlattenedParseTree.Node)
+
+EvaluateParserRequest = _reflection.GeneratedProtocolMessageType('EvaluateParserRequest', (_message.Message,), {
+
+  'ParseResult' : _reflection.GeneratedProtocolMessageType('ParseResult', (_message.Message,), {
+    'DESCRIPTOR' : _EVALUATEPARSERREQUEST_PARSERESULT,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.EvaluateParserRequest.ParseResult)
+    })
+  ,
+  'DESCRIPTOR' : _EVALUATEPARSERREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.EvaluateParserRequest)
+  })
+_sym_db.RegisterMessage(EvaluateParserRequest)
+_sym_db.RegisterMessage(EvaluateParserRequest.ParseResult)
+
+EvaluateParserResponse = _reflection.GeneratedProtocolMessageType('EvaluateParserResponse', (_message.Message,), {
+  'DESCRIPTOR' : _EVALUATEPARSERRESPONSE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.EvaluateParserResponse)
+  })
+_sym_db.RegisterMessage(EvaluateParserResponse)
+
+TsurgeonRequest = _reflection.GeneratedProtocolMessageType('TsurgeonRequest', (_message.Message,), {
+
+  'Operation' : _reflection.GeneratedProtocolMessageType('Operation', (_message.Message,), {
+    'DESCRIPTOR' : _TSURGEONREQUEST_OPERATION,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TsurgeonRequest.Operation)
+    })
+  ,
+  'DESCRIPTOR' : _TSURGEONREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TsurgeonRequest)
+  })
+_sym_db.RegisterMessage(TsurgeonRequest)
+_sym_db.RegisterMessage(TsurgeonRequest.Operation)
+
+TsurgeonResponse = _reflection.GeneratedProtocolMessageType('TsurgeonResponse', (_message.Message,), {
+  'DESCRIPTOR' : _TSURGEONRESPONSE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.TsurgeonResponse)
+  })
+_sym_db.RegisterMessage(TsurgeonResponse)
+
+MorphologyRequest = _reflection.GeneratedProtocolMessageType('MorphologyRequest', (_message.Message,), {
+
+  'TaggedWord' : _reflection.GeneratedProtocolMessageType('TaggedWord', (_message.Message,), {
+    'DESCRIPTOR' : _MORPHOLOGYREQUEST_TAGGEDWORD,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.MorphologyRequest.TaggedWord)
+    })
+  ,
+  'DESCRIPTOR' : _MORPHOLOGYREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.MorphologyRequest)
+  })
+_sym_db.RegisterMessage(MorphologyRequest)
+_sym_db.RegisterMessage(MorphologyRequest.TaggedWord)
+
+MorphologyResponse = _reflection.GeneratedProtocolMessageType('MorphologyResponse', (_message.Message,), {
+
+  'WordTagLemma' : _reflection.GeneratedProtocolMessageType('WordTagLemma', (_message.Message,), {
+    'DESCRIPTOR' : _MORPHOLOGYRESPONSE_WORDTAGLEMMA,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.MorphologyResponse.WordTagLemma)
+    })
+  ,
+  'DESCRIPTOR' : _MORPHOLOGYRESPONSE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.MorphologyResponse)
+  })
+_sym_db.RegisterMessage(MorphologyResponse)
+_sym_db.RegisterMessage(MorphologyResponse.WordTagLemma)
+
+DependencyConverterRequest = _reflection.GeneratedProtocolMessageType('DependencyConverterRequest', (_message.Message,), {
+  'DESCRIPTOR' : _DEPENDENCYCONVERTERREQUEST,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.DependencyConverterRequest)
+  })
+_sym_db.RegisterMessage(DependencyConverterRequest)
+
+DependencyConverterResponse = _reflection.GeneratedProtocolMessageType('DependencyConverterResponse', (_message.Message,), {
+
+  'DependencyConversion' : _reflection.GeneratedProtocolMessageType('DependencyConversion', (_message.Message,), {
+    'DESCRIPTOR' : _DEPENDENCYCONVERTERRESPONSE_DEPENDENCYCONVERSION,
+    '__module__' : 'CoreNLP_pb2'
+    # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.DependencyConverterResponse.DependencyConversion)
+    })
+  ,
+  'DESCRIPTOR' : _DEPENDENCYCONVERTERRESPONSE,
+  '__module__' : 'CoreNLP_pb2'
+  # @@protoc_insertion_point(class_scope:edu.stanford.nlp.pipeline.DependencyConverterResponse)
+  })
+_sym_db.RegisterMessage(DependencyConverterResponse)
+_sym_db.RegisterMessage(DependencyConverterResponse.DependencyConversion)
+
+if _descriptor._USE_C_DESCRIPTORS == False:
+
+  DESCRIPTOR._options = None
+  DESCRIPTOR._serialized_options = b'\n\031edu.stanford.nlp.pipelineB\rCoreNLPProtos'
+  _DEPENDENCYGRAPH.fields_by_name['root']._options = None
+  _DEPENDENCYGRAPH.fields_by_name['root']._serialized_options = b'\020\001'
+  _DEPENDENCYGRAPH.fields_by_name['rootNode']._options = None
+  _DEPENDENCYGRAPH.fields_by_name['rootNode']._serialized_options = b'\020\001'
+  _LANGUAGE._serialized_start=13457
+  _LANGUAGE._serialized_end=13620
+  _SENTIMENT._serialized_start=13622
+  _SENTIMENT._serialized_end=13726
+  _NATURALLOGICRELATION._serialized_start=13729
+  _NATURALLOGICRELATION._serialized_end=13876
+  _DOCUMENT._serialized_start=45
+  _DOCUMENT._serialized_end=782
+  _SENTENCE._serialized_start=785
+  _SENTENCE._serialized_end=2820
+  _TOKEN._serialized_start=2823
+  _TOKEN._serialized_end=4477
+  _QUOTE._serialized_start=4480
+  _QUOTE._serialized_end=4964
+  _PARSETREE._serialized_start=4967
+  _PARSETREE._serialized_end=5166
+  _DEPENDENCYGRAPH._serialized_start=5169
+  _DEPENDENCYGRAPH._serialized_end=5708
+  _DEPENDENCYGRAPH_NODE._serialized_start=5403
+  _DEPENDENCYGRAPH_NODE._serialized_end=5491
+  _DEPENDENCYGRAPH_EDGE._serialized_start=5494
+  _DEPENDENCYGRAPH_EDGE._serialized_end=5708
+  _COREFCHAIN._serialized_start=5711
+  _COREFCHAIN._serialized_end=6037
+  _COREFCHAIN_COREFMENTION._serialized_start=5836
+  _COREFCHAIN_COREFMENTION._serialized_end=6037
+  _MENTION._serialized_start=6040
+  _MENTION._serialized_end=7175
+  _INDEXEDWORD._serialized_start=7177
+  _INDEXEDWORD._serialized_end=7265
+  _SPEAKERINFO._serialized_start=7267
+  _SPEAKERINFO._serialized_end=7319
+  _SPAN._serialized_start=7321
+  _SPAN._serialized_end=7355
+  _TIMEX._serialized_start=7357
+  _TIMEX._serialized_end=7476
+  _ENTITY._serialized_start=7479
+  _ENTITY._serialized_end=7698
+  _RELATION._serialized_start=7701
+  _RELATION._serialized_end=7884
+  _OPERATOR._serialized_start=7887
+  _OPERATOR._serialized_end=8065
+  _POLARITY._serialized_start=8068
+  _POLARITY._serialized_end=8621
+  _NERMENTION._serialized_start=8624
+  _NERMENTION._serialized_end=8973
+  _SENTENCEFRAGMENT._serialized_start=8975
+  _SENTENCEFRAGMENT._serialized_end=9064
+  _TOKENLOCATION._serialized_start=9066
+  _TOKENLOCATION._serialized_end=9124
+  _RELATIONTRIPLE._serialized_start=9127
+  _RELATIONTRIPLE._serialized_end=9537
+  _MAPSTRINGSTRING._serialized_start=9539
+  _MAPSTRINGSTRING._serialized_end=9584
+  _MAPINTSTRING._serialized_start=9586
+  _MAPINTSTRING._serialized_end=9628
+  _SECTION._serialized_start=9631
+  _SECTION._serialized_end=9883
+  _SEMGREXREQUEST._serialized_start=9886
+  _SEMGREXREQUEST._serialized_end=10114
+  _SEMGREXREQUEST_DEPENDENCIES._serialized_start=9992
+  _SEMGREXREQUEST_DEPENDENCIES._serialized_end=10114
+  _SEMGREXRESPONSE._serialized_start=10117
+  _SEMGREXRESPONSE._serialized_end=10880
+  _SEMGREXRESPONSE_NAMEDNODE._serialized_start=10208
+  _SEMGREXRESPONSE_NAMEDNODE._serialized_end=10253
+  _SEMGREXRESPONSE_NAMEDRELATION._serialized_start=10255
+  _SEMGREXRESPONSE_NAMEDRELATION._serialized_end=10298
+  _SEMGREXRESPONSE_NAMEDEDGE._serialized_start=10301
+  _SEMGREXRESPONSE_NAMEDEDGE._serialized_end=10429
+  _SEMGREXRESPONSE_MATCH._serialized_start=10432
+  _SEMGREXRESPONSE_MATCH._serialized_end=10709
+  _SEMGREXRESPONSE_SEMGREXRESULT._serialized_start=10711
+  _SEMGREXRESPONSE_SEMGREXRESULT._serialized_end=10791
+  _SEMGREXRESPONSE_GRAPHRESULT._serialized_start=10793
+  _SEMGREXRESPONSE_GRAPHRESULT._serialized_end=10880
+  _SSURGEONREQUEST._serialized_start=10883
+  _SSURGEONREQUEST._serialized_end=11123
+  _SSURGEONREQUEST_SSURGEON._serialized_start=11032
+  _SSURGEONREQUEST_SSURGEON._serialized_end=11123
+  _SSURGEONRESPONSE._serialized_start=11126
+  _SSURGEONRESPONSE._serialized_end=11314
+  _SSURGEONRESPONSE_SSURGEONRESULT._serialized_start=11222
+  _SSURGEONRESPONSE_SSURGEONRESULT._serialized_end=11314
+  _TOKENSREGEXREQUEST._serialized_start=11316
+  _TOKENSREGEXREQUEST._serialized_end=11403
+  _TOKENSREGEXRESPONSE._serialized_start=11406
+  _TOKENSREGEXRESPONSE._serialized_end=11829
+  _TOKENSREGEXRESPONSE_MATCHLOCATION._serialized_start=11505
+  _TOKENSREGEXRESPONSE_MATCHLOCATION._serialized_end=11562
+  _TOKENSREGEXRESPONSE_MATCH._serialized_start=11565
+  _TOKENSREGEXRESPONSE_MATCH._serialized_end=11744
+  _TOKENSREGEXRESPONSE_PATTERNMATCH._serialized_start=11746
+  _TOKENSREGEXRESPONSE_PATTERNMATCH._serialized_end=11829
+  _DEPENDENCYENHANCERREQUEST._serialized_start=11832
+  _DEPENDENCYENHANCERREQUEST._serialized_end=12006
+  _FLATTENEDPARSETREE._serialized_start=12009
+  _FLATTENEDPARSETREE._serialized_end=12189
+  _FLATTENEDPARSETREE_NODE._serialized_start=12098
+  _FLATTENEDPARSETREE_NODE._serialized_end=12189
+  _EVALUATEPARSERREQUEST._serialized_start=12192
+  _EVALUATEPARSERREQUEST._serialized_end=12438
+  _EVALUATEPARSERREQUEST_PARSERESULT._serialized_start=12298
+  _EVALUATEPARSERREQUEST_PARSERESULT._serialized_end=12438
+  _EVALUATEPARSERRESPONSE._serialized_start=12440
+  _EVALUATEPARSERRESPONSE._serialized_end=12509
+  _TSURGEONREQUEST._serialized_start=12512
+  _TSURGEONREQUEST._serialized_end=12712
+  _TSURGEONREQUEST_OPERATION._serialized_start=12667
+  _TSURGEONREQUEST_OPERATION._serialized_end=12712
+  _TSURGEONRESPONSE._serialized_start=12714
+  _TSURGEONRESPONSE._serialized_end=12794
+  _MORPHOLOGYREQUEST._serialized_start=12797
+  _MORPHOLOGYREQUEST._serialized_end=12930
+  _MORPHOLOGYREQUEST_TAGGEDWORD._serialized_start=12890
+  _MORPHOLOGYREQUEST_TAGGEDWORD._serialized_end=12930
+  _MORPHOLOGYRESPONSE._serialized_start=12933
+  _MORPHOLOGYRESPONSE._serialized_end=13087
+  _MORPHOLOGYRESPONSE_WORDTAGLEMMA._serialized_start=13030
+  _MORPHOLOGYRESPONSE_WORDTAGLEMMA._serialized_end=13087
+  _DEPENDENCYCONVERTERREQUEST._serialized_start=13089
+  _DEPENDENCYCONVERTERREQUEST._serialized_end=13179
+  _DEPENDENCYCONVERTERRESPONSE._serialized_start=13182
+  _DEPENDENCYCONVERTERRESPONSE._serialized_end=13454
+  _DEPENDENCYCONVERTERRESPONSE_DEPENDENCYCONVERSION._serialized_start=13312
+  _DEPENDENCYCONVERTERRESPONSE_DEPENDENCYCONVERSION._serialized_end=13454
+# @@protoc_insertion_point(module_scope)

stanza/stanza/protobuf/__init__.py

@@ -0,0 +1,52 @@
+from __future__ import absolute_import
+
+from io import BytesIO
+import warnings
+
+from google.protobuf.internal.encoder import _EncodeVarint
+from google.protobuf.internal.decoder import _DecodeVarint
+from google.protobuf.message import DecodeError
+from .CoreNLP_pb2 import *
+
+def parseFromDelimitedString(obj, buf, offset=0):
+    """
+    Stanford CoreNLP uses the Java "writeDelimitedTo" function, which
+    writes the size (and offset) of the buffer before writing the object.
+    This function handles parsing this message starting from offset 0.
+
+    @returns how many bytes of @buf were consumed.
+    """
+    size, pos = _DecodeVarint(buf, offset)
+    try:
+        obj.ParseFromString(buf[offset+pos:offset+pos+size])
+    except DecodeError as e:
+        warnings.warn("Failed to decode a serialized output from CoreNLP server. An incomplete or empty object will be returned.", \
+            RuntimeWarning)
+    return pos+size
+
+def writeToDelimitedString(obj, stream=None):
+    """
+    Stanford CoreNLP uses the Java "writeDelimitedTo" function, which
+    writes the size (and offset) of the buffer before writing the object.
+    This function handles parsing this message starting from offset 0.
+
+    @returns how many bytes of @buf were consumed.
+    """
+    if stream is None:
+        stream = BytesIO()
+
+    _EncodeVarint(stream.write, obj.ByteSize(), True)
+    stream.write(obj.SerializeToString())
+    return stream
+
+def to_text(sentence):
+    """
+    Helper routine that converts a Sentence protobuf to a string from
+    its tokens.
+    """
+    text = ""
+    for i, tok in enumerate(sentence.token):
+        if i != 0:
+            text += tok.before
+        text += tok.word
+    return text

stanza/stanza/resources/common.py

@@ -0,0 +1,619 @@
+"""
+Common utilities for Stanza resources.
+"""
+
+from collections import defaultdict, namedtuple
+import errno
+import hashlib
+import json
+import logging
+import os
+from pathlib import Path
+import requests
+import shutil
+import tempfile
+import zipfile
+
+from tqdm.auto import tqdm
+
+from stanza.utils.helper_func import make_table
+from stanza.pipeline._constants import TOKENIZE, MWT, POS, LEMMA, DEPPARSE, NER, SENTIMENT
+from stanza.pipeline.registry import PIPELINE_NAMES, PROCESSOR_VARIANTS
+from stanza.resources.default_packages import PACKAGES
+from stanza._version import __resources_version__
+
+logger = logging.getLogger('stanza')
+
+# set home dir for default
+HOME_DIR = str(Path.home())
+STANFORDNLP_RESOURCES_URL = 'https://nlp.stanford.edu/software/stanza/stanza-resources/'
+STANZA_RESOURCES_GITHUB = 'https://raw.githubusercontent.com/stanfordnlp/stanza-resources/'
+DEFAULT_RESOURCES_URL = os.getenv('STANZA_RESOURCES_URL', STANZA_RESOURCES_GITHUB + 'main')
+DEFAULT_RESOURCES_VERSION = os.getenv(
+    'STANZA_RESOURCES_VERSION',
+    __resources_version__
+)
+DEFAULT_MODEL_URL = os.getenv('STANZA_MODEL_URL', 'default')
+DEFAULT_MODEL_DIR = os.getenv(
+    'STANZA_RESOURCES_DIR',
+    os.path.join(HOME_DIR, 'stanza_resources')
+)
+
+PRETRAIN_NAMES = ("pretrain", "forward_charlm", "backward_charlm")
+
+class ResourcesFileNotFoundError(FileNotFoundError):
+    def __init__(self, resources_filepath):
+        super().__init__(f"Resources file not found at: {resources_filepath}  Try to download the model again.")
+        self.resources_filepath = resources_filepath
+
+class UnknownLanguageError(ValueError):
+    def __init__(self, unknown):
+        super().__init__(f"Unknown language requested: {unknown}")
+        self.unknown_language = unknown
+
+class UnknownProcessorError(ValueError):
+    def __init__(self, unknown):
+        super().__init__(f"Unknown processor type requested: {unknown}")
+        self.unknown_processor = unknown
+
+ModelSpecification = namedtuple('ModelSpecification', ['processor', 'package', 'dependencies'])
+
+def ensure_dir(path):
+    """
+    Create dir in case it does not exist.
+    """
+    Path(path).mkdir(parents=True, exist_ok=True)
+
+def get_md5(path):
+    """
+    Get the MD5 value of a path.
+    """
+    try:
+        with open(path, 'rb') as fin:
+            data = fin.read()
+    except OSError as e:
+        if not e.filename:
+            e.filename = path
+        raise
+    return hashlib.md5(data).hexdigest()
+
+def unzip(path, filename):
+    """
+    Fully unzip a file `filename` that's in a directory `dir`.
+    """
+    logger.debug(f'Unzip: {path}/{filename}...')
+    with zipfile.ZipFile(os.path.join(path, filename)) as f:
+        f.extractall(path)
+
+def get_root_from_zipfile(filename):
+    """
+    Get the root directory from a archived zip file.
+    """
+    zf = zipfile.ZipFile(filename, "r")
+    assert len(zf.filelist) > 0, \
+        f"Zip file at f{filename} seems to be corrupted. Please check it."
+    return os.path.dirname(zf.filelist[0].filename)
+
+def file_exists(path, md5):
+    """
+    Check if the file at `path` exists and match the provided md5 value.
+    """
+    return os.path.exists(path) and get_md5(path) == md5
+
+def assert_file_exists(path, md5=None, alternate_md5=None):
+    if not os.path.exists(path):
+        raise FileNotFoundError(errno.ENOENT, "Cannot find expected file", path)
+    if md5:
+        file_md5 = get_md5(path)
+        if file_md5 != md5:
+            if file_md5 == alternate_md5:
+                logger.debug("Found a possibly older version of file %s, md5 %s instead of %s", path, alternate_md5, md5)
+            else:
+                raise ValueError("md5 for %s is %s, expected %s" % (path, file_md5, md5))
+
+def download_file(url, path, proxies, raise_for_status=False):
+    """
+    Download a URL into a file as specified by `path`.
+    """
+    verbose = logger.level in [0, 10, 20]
+    r = requests.get(url, stream=True, proxies=proxies)
+    if raise_for_status:
+        r.raise_for_status()
+    with open(path, 'wb') as f:
+        file_size = int(r.headers.get('content-length'))
+        default_chunk_size = 131072
+        desc = 'Downloading ' + url
+        with tqdm(total=file_size, unit='B', unit_scale=True, \
+            disable=not verbose, desc=desc) as pbar:
+            for chunk in r.iter_content(chunk_size=default_chunk_size):
+                if chunk:
+                    f.write(chunk)
+                    f.flush()
+                    pbar.update(len(chunk))
+    return r.status_code
+
+def request_file(url, path, proxies=None, md5=None, raise_for_status=False, log_info=True, alternate_md5=None):
+    """
+    A complete wrapper over download_file() that also make sure the directory of
+    `path` exists, and that a file matching the md5 value does not exist.
+
+    alternate_md5 allows for an alternate md5 that is acceptable (such as if an older version of a file is okay)
+    """
+    basedir = Path(path).parent
+    ensure_dir(basedir)
+    if file_exists(path, md5):
+        if log_info:
+            logger.info(f'File exists: {path}')
+        else:
+            logger.debug(f'File exists: {path}')
+        return
+    # We write data first to a temporary directory,
+    # then use os.replace() so that multiple processes
+    # running at the same time don't clobber each other
+    # with partially downloaded files
+    # This was especially common with resources.json
+    with tempfile.TemporaryDirectory(dir=basedir) as temp:
+        temppath = os.path.join(temp, os.path.split(path)[-1])
+        download_file(url, temppath, proxies, raise_for_status)
+        os.replace(temppath, path)
+    assert_file_exists(path, md5, alternate_md5)
+    if log_info:
+        logger.info(f'Downloaded file to {path}')
+    else:
+        logger.debug(f'Downloaded file to {path}')
+
+def sort_processors(processor_list):
+    sorted_list = []
+    for processor in PIPELINE_NAMES:
+        for item in processor_list:
+            if item[0] == processor:
+                sorted_list.append(item)
+    # going just by processors in PIPELINE_NAMES, this drops any names
+    # which are not an official processor but might still be useful
+    # check the list and append them to the end
+    # this is especially useful when downloading pretrain or charlm models
+    for processor in processor_list:
+        for item in sorted_list:
+            if processor[0] == item[0]:
+                break
+        else:
+            sorted_list.append(item)
+    return sorted_list
+
+def add_mwt(processors, resources, lang):
+    """Add mwt if tokenize is passed without mwt.
+
+    If tokenize is in the list, but mwt is not, and there is a corresponding
+    tokenize and mwt pair in the resources file, mwt is added so no missing
+    mwt errors are raised.
+
+    TODO: how does this handle EWT in English?
+    """
+    value = processors[TOKENIZE]
+    if value in resources[lang][PACKAGES] and MWT in resources[lang][PACKAGES][value]:
+        logger.warning("Language %s package %s expects mwt, which has been added", lang, value)
+        processors[MWT] = value
+    elif (value in resources[lang][TOKENIZE] and MWT in resources[lang] and value in resources[lang][MWT]):
+        logger.warning("Language %s package %s expects mwt, which has been added", lang, value)
+        processors[MWT] = value
+
+def maintain_processor_list(resources, lang, package, processors, allow_pretrain=False, maybe_add_mwt=True):
+    """
+    Given a parsed resources file, language, and possible package
+    and/or processors, expands the package to the list of processors
+
+    Returns a list of processors
+    Each item in the list of processors is a pair:
+      name, then a list of ModelSpecification
+    so, for example:
+      [['pos', [ModelSpecification(processor='pos', package='gsd', dependencies=None)]],
+       ['depparse', [ModelSpecification(processor='depparse', package='gsd', dependencies=None)]]]
+    """
+    processor_list = defaultdict(list)
+    # resolve processor models
+    if processors:
+        logger.debug(f'Processing parameter "processors"...')
+        if maybe_add_mwt and TOKENIZE in processors and MWT not in processors:
+            add_mwt(processors, resources, lang)
+        for key, plist in processors.items():
+            if not isinstance(key, str):
+                raise ValueError("Processor names must be strings")
+            if not isinstance(plist, (tuple, list, str)):
+                raise ValueError("Processor values must be strings")
+            if isinstance(plist, str):
+                plist = [plist]
+            if key not in PIPELINE_NAMES:
+                if not allow_pretrain or key not in PRETRAIN_NAMES:
+                    raise UnknownProcessorError(key)
+            for value in plist:
+                # check if keys and values can be found
+                if key in resources[lang] and value in resources[lang][key]:
+                    logger.debug(f'Found {key}: {value}.')
+                    processor_list[key].append(value)
+                # allow values to be default in some cases
+                elif value in resources[lang][PACKAGES] and key in resources[lang][PACKAGES][value]:
+                    logger.debug(
+                        f'Found {key}: {resources[lang][PACKAGES][value][key]}.'
+                    )
+                    processor_list[key].append(resources[lang][PACKAGES][value][key])
+                # optional defaults will be activated if specifically turned on
+                elif value in resources[lang][PACKAGES] and 'optional' in resources[lang][PACKAGES][value] and key in resources[lang][PACKAGES][value]['optional']:
+                    logger.debug(
+                        f"Found {key}: {resources[lang][PACKAGES][value]['optional'][key]}."
+                    )
+                    processor_list[key].append(resources[lang][PACKAGES][value]['optional'][key])
+                # allow processors to be set to variants that we didn't implement
+                elif value in PROCESSOR_VARIANTS[key]:
+                    logger.debug(
+                        f'Found {key}: {value}. '
+                        f'Using external {value} variant for the {key} processor.'
+                    )
+                    processor_list[key].append(value)
+                # allow lemma to be set to "identity"
+                elif key == LEMMA and value == 'identity':
+                    logger.debug(
+                        f'Found {key}: {value}. Using identity lemmatizer.'
+                    )
+                    processor_list[key].append(value)
+                # not a processor in the officially supported processor list
+                elif key not in resources[lang]:
+                    logger.debug(
+                        f'{key}: {value} is not officially supported by Stanza, '
+                        f'loading it anyway.'
+                    )
+                    processor_list[key].append(value)
+                # cannot find the package for a processor and warn user
+                else:
+                    logger.warning(
+                        f'Can not find {key}: {value} from official model list. '
+                        f'Ignoring it.'
+                    )
+    # resolve package
+    if package:
+        logger.debug(f'Processing parameter "package"...')
+        if PACKAGES in resources[lang] and package in resources[lang][PACKAGES]:
+            for key, value in resources[lang][PACKAGES][package].items():
+                if key != 'optional' and key not in processor_list:
+                    logger.debug(f'Found {key}: {value}.')
+                    processor_list[key].append(value)
+        else:
+            flag = False
+            for key in PIPELINE_NAMES:
+                if key not in resources[lang]: continue
+                if package in resources[lang][key]:
+                    flag = True
+                    if key not in processor_list:
+                        logger.debug(f'Found {key}: {package}.')
+                        processor_list[key].append(package)
+                    else:
+                        logger.debug(
+                            f'{key}: {package} is overwritten by '
+                            f'{key}: {processors[key]}.'
+                        )
+            if not flag: logger.warning((f'Can not find package: {package}.'))
+    processor_list = [[key, [ModelSpecification(processor=key, package=value, dependencies=None) for value in plist]] for key, plist in processor_list.items()]
+    processor_list = sort_processors(processor_list)
+    return processor_list
+
+def add_dependencies(resources, lang, processor_list):
+    """
+    Expand the processor_list as given in maintain_processor_list to have the dependencies
+
+    Still a list of model types to ModelSpecifications
+    the dependencies are tuples: name and package
+    for example:
+    [['pos', (ModelSpecification(processor='pos', package='gsd', dependencies=(('pretrain', 'gsd'),)),)],
+     ['depparse', (ModelSpecification(processor='depparse', package='gsd', dependencies=(('pretrain', 'gsd'),)),)]]
+    """
+    lang_resources = resources[lang]
+    for item in processor_list:
+        processor, model_specs = item
+        new_model_specs = []
+        for model_spec in model_specs:
+            # skip dependency checking for external variants of processors and identity lemmatizer
+            if not any([
+                    model_spec.package in PROCESSOR_VARIANTS[processor],
+                    processor == LEMMA and model_spec.package == 'identity'
+                ]):
+                dependencies = lang_resources.get(processor, {}).get(model_spec.package, {}).get('dependencies', [])
+                dependencies = [(dependency['model'], dependency['package']) for dependency in dependencies]
+                model_spec = model_spec._replace(dependencies=tuple(dependencies))
+                logger.debug("Found dependencies %s for processor %s model %s", dependencies, processor, model_spec.package)
+            new_model_specs.append(model_spec)
+        item[1] = tuple(new_model_specs)
+    return processor_list
+
+def flatten_processor_list(processor_list):
+    """
+    The flattened processor list is just a list of types & packages
+
+    For example:
+      [['pos', 'gsd'], ['depparse', 'gsd'], ['pretrain', 'gsd']]
+    """
+    flattened_processor_list = []
+    dependencies_list = []
+    for item in processor_list:
+        processor, model_specs = item
+        for model_spec in model_specs:
+            package = model_spec.package
+            dependencies = model_spec.dependencies
+            flattened_processor_list.append([processor, package])
+            if dependencies:
+                dependencies_list += [tuple(dependency) for dependency in dependencies]
+    dependencies_list = [list(item) for item in set(dependencies_list)]
+    for processor, package in dependencies_list:
+        logger.debug(f'Find dependency {processor}: {package}.')
+    flattened_processor_list += dependencies_list
+    return flattened_processor_list
+
+def set_logging_level(logging_level, verbose):
+    # Check verbose for easy logging control
+    if verbose == False:
+        logging_level = 'ERROR'
+    elif verbose == True:
+        logging_level = 'INFO'
+
+    if logging_level is None:
+        # default logging level of INFO is set in stanza.__init__
+        # but the user may have set it via the logging API
+        # it should NOT be 0, but let's check to be sure...
+        if logger.level == 0:
+            logger.setLevel('INFO')
+        return logger.level
+
+    # Set logging level
+    logging_level = logging_level.upper()
+    all_levels = ['DEBUG', 'INFO', 'WARNING', 'WARN', 'ERROR', 'CRITICAL', 'FATAL']
+    if logging_level not in all_levels:
+        raise ValueError(
+            f"Unrecognized logging level for pipeline: "
+            f"{logging_level}. Must be one of {', '.join(all_levels)}."
+        )
+    logger.setLevel(logging_level)
+    return logger.level
+
+def process_pipeline_parameters(lang, model_dir, package, processors):
+    # Check parameter types and convert values to lower case
+    if isinstance(lang, str):
+        lang = lang.strip().lower()
+    elif lang is not None:
+        raise TypeError(
+            f"The parameter 'lang' should be str, "
+            f"but got {type(lang).__name__} instead."
+        )
+
+    if isinstance(model_dir, str):
+        model_dir = model_dir.strip()
+    elif model_dir is not None:
+        raise TypeError(
+            f"The parameter 'model_dir' should be str, "
+            f"but got {type(model_dir).__name__} instead."
+        )
+
+    if isinstance(processors, (str, list, tuple)):
+        # Special case: processors is str, compatible with older version
+        # also allow for setting alternate packages for these processors
+        # via the package argument
+        if package is None:
+            # each processor will be 'default' for this language
+            package = defaultdict(lambda: 'default')
+        elif isinstance(package, str):
+            # same, but now the named package will be the default instead
+            default = package
+            package = defaultdict(lambda: default)
+        elif isinstance(package, dict):
+            # the dictionary of packages will be used to build the processors dict
+            # any processor not specified in package will be 'default'
+            package = defaultdict(lambda: 'default', package)
+        else:
+            raise TypeError(
+                f"The parameter 'package' should be None, str, or dict, "
+                f"but got {type(package).__name__} instead."
+            )
+        if isinstance(processors, str):
+            processors = [x.strip().lower() for x in processors.split(",")]
+        processors = {
+            processor: package[processor] for processor in processors
+        }
+        package = None
+    elif isinstance(processors, dict):
+        processors = {
+            k.strip().lower(): ([v_i.strip().lower() for v_i in v] if isinstance(v, (tuple, list)) else v.strip().lower())
+            for k, v in processors.items()
+        }
+    elif processors is not None:
+        raise TypeError(
+            f"The parameter 'processors' should be dict or str, "
+            f"but got {type(processors).__name__} instead."
+        )
+
+    if isinstance(package, str):
+        package = package.strip().lower()
+    elif package is not None:
+        raise TypeError(
+            f"The parameter 'package' should be str, or a dict if 'processors' is a str, "
+            f"but got {type(package).__name__} instead."
+        )
+
+    return lang, model_dir, package, processors
+
+def download_resources_json(model_dir=DEFAULT_MODEL_DIR,
+                            resources_url=DEFAULT_RESOURCES_URL,
+                            resources_branch=None,
+                            resources_version=DEFAULT_RESOURCES_VERSION,
+                            resources_filepath=None,
+                            proxies=None):
+    """
+    Downloads resources.json to obtain latest packages.
+    """
+    if resources_url == DEFAULT_RESOURCES_URL and resources_branch is not None:
+        resources_url = STANZA_RESOURCES_GITHUB + resources_branch
+    # handle short name for resources urls; otherwise treat it as url
+    if resources_url.lower() in ('stanford', 'stanfordnlp'):
+        resources_url = STANFORDNLP_RESOURCES_URL
+    resources_url = f'{resources_url}/resources_{resources_version}.json'
+    logger.debug('Downloading resource file from %s', resources_url)
+    if resources_filepath is None:
+        resources_filepath = os.path.join(model_dir, 'resources.json')
+    # make request
+    request_file(
+        resources_url,
+        resources_filepath,
+        proxies,
+        raise_for_status=True
+    )
+
+
+def load_resources_json(model_dir=DEFAULT_MODEL_DIR, resources_filepath=None):
+    """
+    Unpack the resources json file from the given model_dir
+    """
+    if resources_filepath is None:
+        resources_filepath = os.path.join(model_dir, 'resources.json')
+    if not os.path.exists(resources_filepath):
+        raise ResourcesFileNotFoundError(resources_filepath)
+    with open(resources_filepath, encoding="utf-8") as fin:
+        resources = json.load(fin)
+    return resources
+
+def get_language_resources(resources, lang):
+    """
+    Get the resources for a lang from an already loaded resources json, following 'alias' if needed
+    """
+    if lang not in resources:
+        return None
+
+    lang_resources = resources[lang]
+    while 'alias' in lang_resources:
+        lang = lang_resources['alias']
+        lang_resources = resources[lang]
+
+    return lang_resources
+
+def list_available_languages(model_dir=DEFAULT_MODEL_DIR,
+                             resources_url=DEFAULT_RESOURCES_URL,
+                             resources_branch=None,
+                             resources_version=DEFAULT_RESOURCES_VERSION,
+                             proxies=None):
+    """
+    List the non-alias languages in the resources file
+    """
+    download_resources_json(model_dir, resources_url, resources_branch, resources_version, resources_filepath=None, proxies=proxies)
+    resources = load_resources_json(model_dir)
+    # isinstance(str) is because of fields such as "url"
+    # 'alias' is because we want to skip German, alias of de, for example
+    languages = [lang for lang in resources
+                 if not isinstance(resources[lang], str) and 'alias' not in resources[lang]]
+    languages = sorted(languages)
+    return languages
+
+def expand_model_url(resources, model_url):
+    """
+    Returns the url in the resources dict if model_url is default, or returns the model_url
+    """
+    return resources['url'] if model_url.lower() == 'default' else model_url
+
+def download_models(download_list,
+                    resources,
+                    lang,
+                    model_dir=DEFAULT_MODEL_DIR,
+                    resources_version=DEFAULT_RESOURCES_VERSION,
+                    model_url=DEFAULT_MODEL_URL,
+                    proxies=None,
+                    log_info=True):
+    lang_name = resources.get(lang, {}).get('lang_name', lang)
+    download_table = make_table(['Processor', 'Package'], download_list)
+    if log_info:
+        log_msg = logger.info
+    else:
+        log_msg = logger.debug
+    log_msg(
+        f'Downloading these customized packages for language: '
+        f'{lang} ({lang_name})...\n{download_table}'
+    )
+
+    url = expand_model_url(resources, model_url)
+
+    # Download packages
+    for key, value in download_list:
+        try:
+            request_file(
+                url.format(resources_version=resources_version, lang=lang, filename=f"{key}/{value}.pt"),
+                os.path.join(model_dir, lang, key, f'{value}.pt'),
+                proxies,
+                md5=resources[lang][key][value]['md5'],
+                log_info=log_info,
+                alternate_md5=resources[lang][key][value].get('alternate_md5', None)
+            )
+        except KeyError as e:
+            raise ValueError(
+                f'Cannot find the following processor and model name combination: '
+                f'{key}, {value}. Please check if you have provided the correct model name.'
+            ) from e
+
+# main download function
+def download(
+        lang='en',
+        model_dir=DEFAULT_MODEL_DIR,
+        package='default',
+        processors={},
+        logging_level=None,
+        verbose=None,
+        resources_url=DEFAULT_RESOURCES_URL,
+        resources_branch=None,
+        resources_version=DEFAULT_RESOURCES_VERSION,
+        model_url=DEFAULT_MODEL_URL,
+        proxies=None,
+        download_json=True
+    ):
+    # set global logging level
+    set_logging_level(logging_level, verbose)
+    # process different pipeline parameters
+    lang, model_dir, package, processors = process_pipeline_parameters(
+        lang, model_dir, package, processors
+    )
+
+    if download_json or not os.path.exists(os.path.join(model_dir, 'resources.json')):
+        if not download_json:
+            logger.warning("Asked to skip downloading resources.json, but the file does not exist.  Downloading anyway")
+        download_resources_json(model_dir, resources_url, resources_branch, resources_version, resources_filepath=None, proxies=proxies)
+
+    resources = load_resources_json(model_dir)
+    if lang not in resources:
+        raise UnknownLanguageError(lang)
+    if 'alias' in resources[lang]:
+        logger.info(f'"{lang}" is an alias for "{resources[lang]["alias"]}"')
+        lang = resources[lang]['alias']
+    lang_name = resources.get(lang, {}).get('lang_name', lang)
+    url = expand_model_url(resources, model_url)
+
+    # Default: download zipfile and unzip
+    if package == 'default' and (processors is None or len(processors) == 0):
+        logger.info(
+            f'Downloading default packages for language: {lang} ({lang_name}) ...'
+        )
+        # want the URL to become, for example:
+        # https://huggingface.co/stanfordnlp/stanza-af/resolve/v1.3.0/models/default.zip
+        # so we hopefully start from
+        # https://huggingface.co/stanfordnlp/stanza-{lang}/resolve/v{resources_version}/models/{filename}
+        request_file(
+            url.format(resources_version=resources_version, lang=lang, filename="default.zip"),
+            os.path.join(model_dir, lang, f'default.zip'),
+            proxies,
+            md5=resources[lang]['default_md5'],
+        )
+        unzip(os.path.join(model_dir, lang), 'default.zip')
+    # Customize: maintain download list
+    else:
+        download_list = maintain_processor_list(resources, lang, package, processors, allow_pretrain=True)
+        download_list = add_dependencies(resources, lang, download_list)
+        download_list = flatten_processor_list(download_list)
+        download_models(download_list=download_list,
+                        resources=resources,
+                        lang=lang,
+                        model_dir=model_dir,
+                        resources_version=resources_version,
+                        model_url=model_url,
+                        proxies=proxies,
+                        log_info=True)
+    logger.info(f'Finished downloading models and saved to {model_dir}')

stanza/stanza/resources/default_packages.py

@@ -0,0 +1,909 @@
+"""
+Constants for default packages, default pretrains, charlms, etc
+
+Separated from prepare_resources.py so that other modules can use the
+same lists / maps without importing the resources script and possibly
+causing a circular import
+"""
+
+import copy
+
+# all languages will have a map which represents the available packages
+PACKAGES = "packages"
+
+# default treebank for languages
+default_treebanks = {
+    "af":      "afribooms",
+    # currently not publicly released!  sent to us from the group developing this resource
+    "ang":     "nerthus",
+    "ar":      "padt",
+    "be":      "hse",
+    "bg":      "btb",
+    "bxr":     "bdt",
+    "ca":      "ancora",
+    "cop":     "scriptorium",
+    "cs":      "pdt",
+    "cu":      "proiel",
+    "cy":      "ccg",
+    "da":      "ddt",
+    "de":      "gsd",
+    "el":      "gdt",
+    "en":      "combined",
+    "es":      "combined",
+    "et":      "edt",
+    "eu":      "bdt",
+    "fa":      "perdt",
+    "fi":      "tdt",
+    "fo":      "farpahc",
+    "fr":      "combined",
+    "fro":     "profiterole",
+    "ga":      "idt",
+    "gd":      "arcosg",
+    "gl":      "ctg",
+    "got":     "proiel",
+    "grc":     "perseus",
+    "gv":      "cadhan",
+    "hbo":     "ptnk",
+    "he":      "combined",
+    "hi":      "hdtb",
+    "hr":      "set",
+    "hsb":     "ufal",
+    "hu":      "szeged",
+    "hy":      "armtdp",
+    "hyw":     "armtdp",
+    "id":      "gsd",
+    "is":      "icepahc",
+    "it":      "combined",
+    "ja":      "gsd",
+    "ka":      "glc",
+    "kk":      "ktb",
+    "kmr":     "mg",
+    "ko":      "kaist",
+    "kpv":     "lattice",
+    "ky":      "ktmu",
+    "la":      "ittb",
+    "lij":     "glt",
+    "lt":      "alksnis",
+    "lv":      "lvtb",
+    "lzh":     "kyoto",
+    "mr":      "ufal",
+    "mt":      "mudt",
+    "my":      "ucsy",
+    "myv":     "jr",
+    "nb":      "bokmaal",
+    "nds":     "lsdc",
+    "nl":      "alpino",
+    "nn":      "nynorsk",
+    "olo":     "kkpp",
+    "orv":     "torot",
+    "ota":     "boun",
+    "pcm":     "nsc",
+    "pl":      "pdb",
+    "pt":      "bosque",
+    "qaf":     "arabizi",
+    "qpm":     "philotis",
+    "qtd":     "sagt",
+    "ro":      "rrt",
+    "ru":      "syntagrus",
+    "sa":      "vedic",
+    "sd":      "isra",
+    "sk":      "snk",
+    "sl":      "ssj",
+    "sme":     "giella",
+    "sq":      "combined",
+    "sr":      "set",
+    "sv":      "talbanken",
+    "swl":     "sslc",
+    "ta":      "ttb",
+    "te":      "mtg",
+    "th":      "orchid",
+    "tr":      "imst",
+    "ug":      "udt",
+    "uk":      "iu",
+    "ur":      "udtb",
+    "vi":      "vtb",
+    "wo":      "wtb",
+    "xcl":     "caval",
+    "zh-hans": "gsdsimp",
+    "zh-hant": "gsd",
+    "multilingual": "ud"
+}
+
+no_pretrain_languages = set([
+    "cop",
+    "olo",
+    "orv",
+    "pcm",
+    "qaf",   # the QAF treebank is code switched and Romanized, so not easy to reuse existing resources
+    "qpm",   # have talked about deriving this from a language neighborinig to Pomak, but that hasn't happened yet
+    "qtd",
+    "swl",
+
+    "multilingual", # special case so that all languages with a default treebank are represented somewhere
+])
+
+
+# in some cases, we give the pretrain a name other than the original
+# name for the UD dataset
+# we will eventually do this for all of the pretrains
+specific_default_pretrains = {
+    "af":      "fasttextwiki",
+    "ang":     "nerthus",
+    "ar":      "conll17",
+    "be":      "fasttextwiki",
+    "bg":      "conll17",
+    "bxr":     "fasttextwiki",
+    "ca":      "conll17",
+    "cs":      "conll17",
+    "cu":      "conll17",
+    "cy":      "fasttext157",
+    "da":      "conll17",
+    "de":      "conll17",
+    "el":      "conll17",
+    "en":      "conll17",
+    "es":      "conll17",
+    "et":      "conll17",
+    "eu":      "conll17",
+    "fa":      "conll17",
+    "fi":      "conll17",
+    "fo":      "fasttextwiki",
+    "fr":      "conll17",
+    "fro":     "conll17",
+    "ga":      "conll17",
+    "gd":      "fasttextwiki",
+    "gl":      "conll17",
+    "got":     "fasttextwiki",
+    "grc":     "conll17",
+    "gv":      "fasttext157",
+    "hbo":     "utah",
+    "he":      "conll17",
+    "hi":      "conll17",
+    "hr":      "conll17",
+    "hsb":     "fasttextwiki",
+    "hu":      "conll17",
+    "hy":      "isprasglove",
+    "hyw":     "isprasglove",
+    "id":      "conll17",
+    "is":      "fasttext157",
+    "it":      "conll17",
+    "ja":      "conll17",
+    "ka":      "fasttext157",
+    "kk":      "fasttext157",
+    "kmr":     "fasttextwiki",
+    "ko":      "conll17",
+    "kpv":     "fasttextwiki",
+    "ky":      "fasttext157",
+    "la":      "conll17",
+    "lij":     "fasttextwiki",
+    "lt":      "fasttextwiki",
+    "lv":      "conll17",
+    "lzh":     "fasttextwiki",
+    "mr":      "fasttextwiki",
+    "mt":      "fasttextwiki",
+    "my":      "ucsy",
+    "myv":     "mokha",
+    "nb":      "conll17",
+    "nds":     "fasttext157",
+    "nl":      "conll17",
+    "nn":      "conll17",
+    "ota":     "conll17",
+    "pl":      "conll17",
+    "pt":      "conll17",
+    "ro":      "conll17",
+    "ru":      "conll17",
+    "sa":      "fasttext157",
+    "sd":      "isra",
+    "sk":      "conll17",
+    "sl":      "conll17",
+    "sme":     "fasttextwiki",
+    "sq":      "fasttext157",
+    "sr":      "fasttextwiki",
+    "sv":      "conll17",
+    "ta":      "fasttextwiki",
+    "te":      "fasttextwiki",
+    "th":      "fasttext157",
+    "tr":      "conll17",
+    "ug":      "conll17",
+    "uk":      "conll17",
+    "ur":      "conll17",
+    "vi":      "conll17",
+    "wo":      "fasttextwiki",
+    "xcl":     "caval",
+    "zh-hans": "fasttext157",
+    "zh-hant": "conll17",
+}
+
+
+def build_default_pretrains(default_treebanks):
+    default_pretrains = dict(default_treebanks)
+    for lang in no_pretrain_languages:
+        default_pretrains.pop(lang, None)
+    for lang in specific_default_pretrains.keys():
+        default_pretrains[lang] = specific_default_pretrains[lang]
+    return default_pretrains
+
+default_pretrains = build_default_pretrains(default_treebanks)
+
+pos_pretrains = {
+    "en": {
+        "craft":            "biomed",
+        "genia":            "biomed",
+        "mimic":            "mimic",
+    },
+}
+
+depparse_pretrains = pos_pretrains
+
+ner_pretrains = {
+    "ar": {
+        "aqmar": "fasttextwiki",
+    },
+    "de": {
+        "conll03":      "fasttextwiki",
+        # the bert version of germeval uses the smaller vector file
+        "germeval2014": "fasttextwiki",
+    },
+    "en": {
+        "anatem":       "biomed",
+        "bc4chemd":     "biomed",
+        "bc5cdr":       "biomed",
+        "bionlp13cg":   "biomed",
+        "jnlpba":       "biomed",
+        "linnaeus":     "biomed",
+        "ncbi_disease": "biomed",
+        "s800":         "biomed",
+
+        "ontonotes":    "fasttextcrawl",
+        # the stanza-train sample NER model should use the default NER pretrain
+        # for English, that is the same as ontonotes
+        "sample":       "fasttextcrawl",
+
+        "conll03":      "glove",
+
+        "i2b2":         "mimic",
+        "radiology":    "mimic",
+    },
+    "es": {
+        "ancora":  "fasttextwiki",
+        "conll02": "fasttextwiki",
+    },
+    "nl": {
+        "conll02": "fasttextwiki",
+        "wikiner": "fasttextwiki",
+    },
+    "ru": {
+        "wikiner": "fasttextwiki",
+    },
+    "th": {
+        "lst20": "fasttext157",
+    },
+}
+
+
+# default charlms for languages
+default_charlms = {
+    "af": "oscar",
+    "ang": "nerthus1024",
+    "ar": "ccwiki",
+    "bg": "conll17",
+    "da": "oscar",
+    "de": "newswiki",
+    "en": "1billion",
+    "es": "newswiki",
+    "fa": "conll17",
+    "fi": "conll17",
+    "fr": "newswiki",
+    "he": "oscar",
+    "hi": "oscar",
+    "id": "oscar2023",
+    "it": "conll17",
+    "ja": "conll17",
+    "kk": "oscar",
+    "mr": "l3cube",
+    "my": "oscar",
+    "nb": "conll17",
+    "nl": "ccwiki",
+    "pl": "oscar",
+    "pt": "oscar2023",
+    "ru": "newswiki",
+    "sd": "isra",
+    "sv": "conll17",
+    "te": "oscar2022",
+    "th": "oscar",
+    "tr": "conll17",
+    "uk": "conll17",
+    "vi": "conll17",
+    "zh-hans": "gigaword"
+}
+
+pos_charlms = {
+    "en": {
+        # none of the English charlms help with craft or genia
+        "craft": None,
+        "genia": None,
+        "mimic": "mimic",
+    },
+    "tr": {   # no idea why, but this particular one goes down in dev score
+        "boun": None,
+    },
+}
+
+depparse_charlms = copy.deepcopy(pos_charlms)
+
+lemma_charlms = copy.deepcopy(pos_charlms)
+
+ner_charlms = {
+    "en": {
+        "conll03": "1billion",
+        "ontonotes": "1billion",
+        "anatem": "pubmed",
+        "bc4chemd": "pubmed",
+        "bc5cdr": "pubmed",
+        "bionlp13cg": "pubmed",
+        "i2b2": "mimic",
+        "jnlpba": "pubmed",
+        "linnaeus": "pubmed",
+        "ncbi_disease": "pubmed",
+        "radiology": "mimic",
+        "s800": "pubmed",
+    },
+    "hu": {
+        "combined": None,
+    },
+    "nn": {
+        "norne": None,
+    },
+}
+
+# default ner for languages
+default_ners = {
+    "af": "nchlt",
+    "ar": "aqmar_charlm",
+    "bg": "bsnlp19",
+    "da": "ddt",
+    "de": "germeval2014",
+    "en": "ontonotes-ww-multi_charlm",
+    "es": "conll02",
+    "fa": "arman",
+    "fi": "turku",
+    "fr": "wikinergold_charlm",
+    "he": "iahlt_charlm",
+    "hu": "combined",
+    "hy": "armtdp",
+    "it": "fbk",
+    "ja": "gsd",
+    "kk": "kazNERD",
+    "mr": "l3cube",
+    "my": "ucsy",
+    "nb": "norne",
+    "nl": "conll02",
+    "nn": "norne",
+    "pl": "nkjp",
+    "ru": "wikiner",
+    "sd": "siner",
+    "sv": "suc3shuffle",
+    "th": "lst20",
+    "tr": "starlang",
+    "uk": "languk",
+    "vi": "vlsp",
+    "zh-hans": "ontonotes",
+}
+
+# a few languages have sentiment classifier models
+default_sentiment = {
+    "en": "sstplus_charlm",
+    "de": "sb10k_charlm",
+    "es": "tass2020_charlm",
+    "mr": "l3cube_charlm",
+    "vi": "vsfc_charlm",
+    "zh-hans": "ren_charlm",
+}
+
+# also, a few languages (very few, currently) have constituency parser models
+default_constituency = {
+    "da": "arboretum_charlm",
+    "de": "spmrl_charlm",
+    "en": "ptb3-revised_charlm",
+    "es": "combined_charlm",
+    "id": "icon_charlm",
+    "it": "vit_charlm",
+    "ja": "alt_charlm",
+    "pt": "cintil_charlm",
+    #"tr": "starlang_charlm",
+    "vi": "vlsp22_charlm",
+    "zh-hans": "ctb-51_charlm",
+}
+
+optional_constituency = {
+    "tr": "starlang_charlm",
+}
+
+# an alternate tokenizer for languages which aren't trained from a base UD source
+default_tokenizer = {
+    "my": "alt",
+}
+
+# ideally we would have a less expensive model as the base model
+#default_coref = {
+#    "en": "ontonotes_roberta-large_finetuned",
+#}
+
+optional_coref = {
+    "ca": "udcoref_xlm-roberta-lora",
+    "cs": "udcoref_xlm-roberta-lora",
+    "de": "udcoref_xlm-roberta-lora",
+    "en": "udcoref_xlm-roberta-lora",
+    "es": "udcoref_xlm-roberta-lora",
+    "fr": "udcoref_xlm-roberta-lora",
+    "hi": "deeph_muril-large-cased-lora",
+    # UD Coref has both nb and nn datasets for Norwegian
+    "nb": "udcoref_xlm-roberta-lora",
+    "nn": "udcoref_xlm-roberta-lora",
+    "pl": "udcoref_xlm-roberta-lora",
+    "ru": "udcoref_xlm-roberta-lora",
+}
+
+"""
+default transformers to use for various languages
+
+we try to document why we choose a particular model in each case
+"""
+TRANSFORMERS = {
+    # We tested three candidate AR models on POS, Depparse, and NER
+    #
+    # POS: padt dev set scores, AllTags
+    # depparse: padt dev set scores, LAS
+    # NER: dev scores on a random split of AQMAR, entity scores
+    #
+    #                                             pos   depparse  ner
+    # none (pt & charlm only)                    94.08    83.49  84.19
+    # asafaya/bert-base-arabic                   95.10    84.96  85.98
+    # aubmindlab/bert-base-arabertv2             95.33    85.28  84.93
+    # aubmindlab/araelectra-base-discriminator   95.66    85.83  86.10
+    "ar": "aubmindlab/araelectra-base-discriminator",
+
+    # https://huggingface.co/Maltehb/danish-bert-botxo
+    # contrary to normal expectations, this hurts F1
+    # on a dev split by about 1 F1
+    # "da": "Maltehb/danish-bert-botxo",
+    #
+    # the multilingual bert is a marginal improvement for conparse
+    #
+    # December 2022 update:
+    # there are quite a few Danish transformers available on HuggingFace
+    # here are the results of training a constituency parser with adadelta/adamw
+    # on each of them:
+    #
+    # no bert                              0.8245    0.8230
+    # alexanderfalk/danbert-small-cased    0.8236    0.8286
+    # Geotrend/distilbert-base-da-cased    0.8268    0.8306
+    # sarnikowski/convbert-small-da-cased  0.8322    0.8341
+    # bert-base-multilingual-cased         0.8341    0.8342
+    # vesteinn/ScandiBERT-no-faroese       0.8373    0.8408
+    # Maltehb/danish-bert-botxo            0.8383    0.8408
+    # vesteinn/ScandiBERT                  0.8421    0.8475
+    #
+    # Also, two models have token windows too short for use with the
+    # Danish dataset:
+    #  jonfd/electra-small-nordic
+    #  Maltehb/aelaectra-danish-electra-small-cased
+    #
+    "da": "vesteinn/ScandiBERT",
+
+    # As of April 2022, the bert models available have a weird
+    # tokenizer issue where soft hyphen causes it to crash.
+    # We attempt to compensate for that in the dev branch
+    #
+    # NER scores
+    #     model                                       dev      text
+    # xlm-roberta-large                              86.56    85.23
+    # bert-base-german-cased                         87.59    86.95
+    # dbmdz/bert-base-german-cased                   88.27    87.47
+    # german-nlp-group/electra-base-german-uncased   88.60    87.09
+    #
+    # constituency scores w/ peft, March 2024 model, in-order
+    #    model             dev     test
+    #   xlm-roberta-base  95.17   93.34
+    #   xlm-roberta-large 95.86   94.46    (!!!)
+    #   bert-base         95.24   93.24
+    #   dbmdz/bert        95.32   93.33
+    #   german/electra    95.72   94.05
+    #
+    # POS scores
+    #    model             dev     test
+    #   None              88.65   87.28
+    #   xlm-roberta-large 89.21   88.11
+    #   bert-base         89.52   88.42
+    #   dbmdz/bert        89.67   88.54
+    #   german/electra    89.98   88.66
+    #
+    # depparse scores, LAS
+    #    model             dev     test
+    #   None              87.76   84.37
+    #   xlm-roberta-large 89.00   85.79
+    #   bert-base         88.72   85.40
+    #   dbmdz/bert        88.70   85.14
+    #   german/electra    89.21   86.06
+    "de": "german-nlp-group/electra-base-german-uncased",
+
+    # experiments on various forms of roberta & electra
+    #  https://huggingface.co/roberta-base
+    #  https://huggingface.co/roberta-large
+    #  https://huggingface.co/google/electra-small-discriminator
+    #  https://huggingface.co/google/electra-base-discriminator
+    #  https://huggingface.co/google/electra-large-discriminator
+    #
+    # experiments using the different models for POS tagging,
+    # dev set, including WV and charlm, AllTags score:
+    #  roberta-base:   95.67
+    #  roberta-large:  95.98
+    #  electra-small:  95.31
+    #  electra-base:   95.90
+    #  electra-large:  96.01
+    #
+    # depparse scores, dev set, no finetuning, with WV and charlm
+    #                   UAS   LAS  CLAS  MLAS  BLEX
+    #  roberta-base:   93.16 91.20 89.87 89.38 89.87
+    #  roberta-large:  93.47 91.56 90.13 89.71 90.13
+    #  electra-small:  92.17 90.02 88.25 87.66 88.25
+    #  electra-base:   93.42 91.44 90.10 89.67 90.10
+    #  electra-large:  94.07 92.17 90.99 90.53 90.99
+    #
+    # conparse scores, dev & test set, with WV and charlm
+    #  roberta_base:   96.05 95.60
+    #  roberta_large:  95.95 95.60
+    #  electra-small:  95.33 95.04
+    #  electra-base:   96.09 95.98
+    #  electra-large:  96.25 96.14
+    #
+    # conparse scores w/ finetune, dev & test set, with WV and charlm
+    #  roberta_base:   96.07 95.81
+    #  roberta_large:  96.37 96.41   (!!!)
+    #  electra-small:  95.62 95.36
+    #  electra-base:   96.21 95.94
+    #  electra-large:  96.40 96.32
+    #
+    "en": "google/electra-large-discriminator",
+
+    # TODO need to test, possibly compare with others
+    "es": "bertin-project/bertin-roberta-base-spanish",
+
+    # NER scores for a couple Persian options:
+    # none:
+    # dev:  2022-04-23 01:44:53 INFO: fa_arman 79.46
+    # test: 2022-04-23 01:45:03 INFO: fa_arman 80.06
+    #
+    # HooshvareLab/bert-fa-zwnj-base
+    # dev:  2022-04-23 02:43:44 INFO: fa_arman 80.87
+    # test: 2022-04-23 02:44:07 INFO: fa_arman 80.81
+    #
+    # HooshvareLab/roberta-fa-zwnj-base
+    # dev:  2022-04-23 16:23:25 INFO: fa_arman 81.23
+    # test: 2022-04-23 16:23:48 INFO: fa_arman 81.11
+    #
+    # HooshvareLab/bert-base-parsbert-uncased
+    # dev:  2022-04-26 10:42:09 INFO: fa_arman 82.49
+    # test: 2022-04-26 10:42:31 INFO: fa_arman 83.16
+    "fa": 'HooshvareLab/bert-base-parsbert-uncased',
+
+    # NER scores for a couple options:
+    # none:
+    # dev:  2022-03-04 INFO: fi_turku 83.45
+    # test: 2022-03-04 INFO: fi_turku 86.25
+    #
+    # bert-base-multilingual-cased
+    # dev:  2022-03-04 INFO: fi_turku 85.23
+    # test: 2022-03-04 INFO: fi_turku 89.00
+    #
+    # TurkuNLP/bert-base-finnish-cased-v1:
+    # dev:  2022-03-04 INFO: fi_turku 88.41
+    # test: 2022-03-04 INFO: fi_turku 91.36
+    "fi": "TurkuNLP/bert-base-finnish-cased-v1",
+
+    # POS dev set tagging results for French:
+    #  No bert:
+    #    98.60  100.00   98.55   98.04
+    #  dbmdz/electra-base-french-europeana-cased-discriminator
+    #    98.70  100.00   98.69   98.24
+    #  benjamin/roberta-base-wechsel-french
+    #    98.71  100.00   98.75   98.26
+    #  camembert/camembert-large
+    #    98.75  100.00   98.75   98.30
+    #  camembert-base
+    #    98.78  100.00   98.77   98.33
+    #
+    # GSD depparse dev set results for French:
+    #  No bert:
+    #    95.83 94.52 91.34 91.10 91.34
+    #  camembert/camembert-large
+    #    96.80 95.71 93.37 93.13 93.37
+    #  TODO: the rest of the chart
+    "fr": "camembert/camembert-large",
+
+    # Ancient Greek has a surprising number of transformers, considering
+    #    Model           POS        Depparse LAS
+    # None              0.8812       0.7684
+    # Microbert M       0.8883       0.7706
+    # Microbert MX      0.8910       0.7755
+    # Microbert MXP     0.8916       0.7742
+    # Pranaydeeps Bert  0.9139       0.7987
+    "grc": "pranaydeeps/Ancient-Greek-BERT",
+
+    # a couple possibilities to experiment with for Hebrew
+    # dev scores for POS and depparse
+    # https://huggingface.co/imvladikon/alephbertgimmel-base-512
+    #   UPOS    XPOS  UFeats AllTags
+    #  97.25   97.25   92.84   91.81
+    #   UAS   LAS  CLAS  MLAS  BLEX
+    #  94.42 92.47 89.49 88.82 89.49
+    #
+    # https://huggingface.co/onlplab/alephbert-base
+    #   UPOS    XPOS  UFeats AllTags
+    #  97.37   97.37   92.50   91.55
+    #   UAS   LAS  CLAS  MLAS  BLEX
+    #  94.06 92.12 88.80 88.13 88.80
+    #
+    # https://huggingface.co/avichr/heBERT
+    #   UPOS    XPOS  UFeats AllTags
+    #  97.09   97.09   92.36   91.28
+    #   UAS   LAS  CLAS  MLAS  BLEX
+    #  94.29 92.30 88.99 88.38 88.99
+    "he": "imvladikon/alephbertgimmel-base-512",
+
+    # can also experiment with xlm-roberta
+    # on a coref dataset from IITH, span F1:
+    #                         dev      test
+    #  xlm-roberta-large   0.63635   0.66579
+    #  muril-large         0.65369   0.68290
+    "hi": "google/muril-large-cased",
+
+    # https://huggingface.co/xlm-roberta-base
+    # Scores by entity for armtdp NER on 18 labels:
+    # no bert : 86.68
+    # xlm-roberta-base : 89.31
+    "hy": "xlm-roberta-base",
+
+    # Indonesian POS experiments: dev set of GSD
+    # python3 stanza/utils/training/run_pos.py id_gsd --no_bert
+    # python3 stanza/utils/training/run_pos.py id_gsd --bert_model ...
+    # also ran on the ICON constituency dataset
+    #  model                                      POS       CON
+    # no_bert                                    89.95     84.74
+    # flax-community/indonesian-roberta-large    89.78 (!)  xxx
+    # flax-community/indonesian-roberta-base     90.14      xxx
+    # indobenchmark/indobert-base-p2             90.09
+    # indobenchmark/indobert-base-p1             90.14
+    # indobenchmark/indobert-large-p1            90.19
+    # indolem/indobert-base-uncased              90.21     88.60
+    # cahya/bert-base-indonesian-1.5G            90.32     88.15
+    # cahya/roberta-base-indonesian-1.5G         90.40     87.27
+    "id": "indolem/indobert-base-uncased",
+
+    # from https://github.com/idb-ita/GilBERTo
+    # annoyingly, it doesn't handle cased text
+    # supposedly there is an argument "do_lower_case"
+    # but that still leaves a lot of unk tokens
+    # "it": "idb-ita/gilberto-uncased-from-camembert",
+    #
+    # from https://github.com/musixmatchresearch/umberto
+    # on NER, this gets 88.37 dev and 91.02 test
+    # another option is dbmdz/bert-base-italian-cased,
+    # which gets 87.27 dev and 90.32 test
+    #
+    #  in-order constituency parser on the VIT dev set:
+    # dbmdz/bert-base-italian-cased                       0.8079
+    # dbmdz/bert-base-italian-xxl-cased:                  0.8195
+    # Musixmatch/umberto-commoncrawl-cased-v1:            0.8256
+    # dbmdz/electra-base-italian-xxl-cased-discriminator: 0.8314
+    #
+    #  FBK NER dev set:
+    # dbmdz/bert-base-italian-cased:                      87.76
+    # Musixmatch/umberto-commoncrawl-cased-v1:            88.62
+    # dbmdz/bert-base-italian-xxl-cased:                  88.84
+    # dbmdz/electra-base-italian-xxl-cased-discriminator: 89.91
+    #
+    #  combined UD POS dev set:                             UPOS    XPOS  UFeats AllTags
+    # dbmdz/bert-base-italian-cased:                       98.62   98.53   98.06   97.49
+    # dbmdz/bert-base-italian-xxl-cased:                   98.61   98.54   98.07   97.58
+    # dbmdz/electra-base-italian-xxl-cased-discriminator:  98.64   98.54   98.14   97.61
+    # Musixmatch/umberto-commoncrawl-cased-v1:             98.56   98.45   98.13   97.62
+    "it": "dbmdz/electra-base-italian-xxl-cased-discriminator",
+
+    # for Japanese
+    # there are others that would also work,
+    # but they require different tokenizers instead of being
+    # plug & play
+    #
+    # Constitutency scores on ALT (in-order)
+    # no bert: 90.68 dev, 91.40 test
+    # rinna:   91.54 dev, 91.89 test
+    "ja": "rinna/japanese-roberta-base",
+
+    # could also try:
+    # l3cube-pune/marathi-bert-v2
+    #  or
+    # https://huggingface.co/l3cube-pune/hindi-marathi-dev-roberta
+    # l3cube-pune/hindi-marathi-dev-roberta
+    #
+    # depparse ufal dev scores:
+    #  no transformer              74.89 63.70 57.43 53.01 57.43
+    #  l3cube-pune/marathi-roberta 76.48 66.21 61.20 57.60 61.20
+    "mr": "l3cube-pune/marathi-roberta",
+
+    # https://huggingface.co/allegro/herbert-base-cased
+    # Scores by entity on the NKJP NER task:
+    # no bert (dev/test): 88.64/88.75
+    # herbert-base-cased (dev/test): 91.48/91.02,
+    # herbert-large-cased (dev/test): 92.25/91.62
+    # sdadas/polish-roberta-large-v2 (dev/test): 92.66/91.22
+    "pl": "allegro/herbert-base-cased",
+
+    # experiments on the cintil conparse dataset
+    # ran a variety of transformer settings
+    # found the following dev set scores after 400 iterations:
+    # Geotrend/distilbert-base-pt-cased : not plug & play
+    # no bert: 0.9082
+    # xlm-roberta-base: 0.9109
+    # xlm-roberta-large: 0.9254
+    # adalbertojunior/distilbert-portuguese-cased: 0.9300
+    # neuralmind/bert-base-portuguese-cased: 0.9307
+    # neuralmind/bert-large-portuguese-cased: 0.9343
+    "pt": "neuralmind/bert-large-portuguese-cased",
+
+    # hope is actually to build our own using a large text collection
+    "sd": "google/muril-large-cased",
+
+    # Tamil options: quite a few, need to run a bunch of experiments
+    #                               dev pos    dev depparse las
+    # no transformer                 82.82        69.12
+    # ai4bharat/indic-bert           82.98        70.47
+    # lgessler/microbert-tamil-mxp   83.21        69.28
+    # monsoon-nlp/tamillion          83.37        69.28
+    # l3cube-pune/tamil-bert         85.27        72.53
+    # d42kw01f/Tamil-RoBERTa         85.59        70.55
+    # google/muril-base-cased        85.67        72.68
+    # google/muril-large-cased       86.30        72.45
+    "ta": "google/muril-large-cased",
+
+
+    # https://huggingface.co/dbmdz/bert-base-turkish-128k-cased
+    # helps the Turkish model quite a bit
+    "tr": "dbmdz/bert-base-turkish-128k-cased",
+
+    # from https://github.com/VinAIResearch/PhoBERT
+    # "vi": "vinai/phobert-base",
+    # using 6 or 7 layers of phobert-large is slightly
+    # more effective for constituency parsing than
+    # using 4 layers of phobert-base
+    # ... going beyond 4 layers of phobert-base
+    # does not help the scores
+    "vi": "vinai/phobert-large",
+
+    # https://github.com/ymcui/Chinese-BERT-wwm
+    # there's also hfl/chinese-roberta-wwm-ext-large
+    # or hfl/chinese-electra-base-discriminator
+    # or hfl/chinese-electra-180g-large-discriminator,
+    #   which works better than the below roberta on constituency
+    # "zh-hans": "hfl/chinese-roberta-wwm-ext",
+    "zh-hans": "hfl/chinese-electra-180g-large-discriminator",
+}
+
+TRANSFORMER_LAYERS = {
+    # not clear what the best number is without more experiments,
+    # but more than 4 is working better than just 4
+    "vi": 7,
+}
+
+TRANSFORMER_NICKNAMES = {
+    # ar
+    "asafaya/bert-base-arabic": "asafaya-bert",
+    "aubmindlab/araelectra-base-discriminator": "aubmind-electra",
+    "aubmindlab/bert-base-arabertv2": "aubmind-bert",
+
+    # da
+    "vesteinn/ScandiBERT": "scandibert",
+
+    # de
+    "bert-base-german-cased": "bert-base-german-cased",
+    "dbmdz/bert-base-german-cased": "dbmdz-bert-german-cased",
+    "german-nlp-group/electra-base-german-uncased": "german-nlp-electra",
+
+    # en
+    "bert-base-multilingual-cased": "mbert",
+    "xlm-roberta-large": "xlm-roberta-large",
+    "google/electra-large-discriminator": "electra-large",
+    "microsoft/deberta-v3-large": "deberta-v3-large",
+
+    # es
+    "bertin-project/bertin-roberta-base-spanish": "bertin-roberta",
+
+    # fa
+    "HooshvareLab/bert-base-parsbert-uncased": "parsbert",
+
+    # fi
+    "TurkuNLP/bert-base-finnish-cased-v1": "bert",
+
+    # fr
+    "benjamin/roberta-base-wechsel-french": "wechsel-roberta",
+    "camembert-base": "camembert-base",
+    "camembert/camembert-large": "camembert-large",
+    "dbmdz/electra-base-french-europeana-cased-discriminator": "dbmdz-electra",
+
+    # grc
+    "pranaydeeps/Ancient-Greek-BERT": "grc-pranaydeeps",
+    "lgessler/microbert-ancient-greek-m": "grc-microbert-m",
+    "lgessler/microbert-ancient-greek-mx": "grc-microbert-mx",
+    "lgessler/microbert-ancient-greek-mxp": "grc-microbert-mxp",
+    "altsoph/bert-base-ancientgreek-uncased": "grc-altsoph",
+
+    # he
+    "imvladikon/alephbertgimmel-base-512" : "alephbertgimmel",
+
+    # hy
+    "xlm-roberta-base": "xlm-roberta-base",
+
+    # id
+    "indolem/indobert-base-uncased":         "indobert",
+    "indobenchmark/indobert-large-p1":       "indobenchmark-large-p1",
+    "indobenchmark/indobert-base-p1":        "indobenchmark-base-p1",
+    "indobenchmark/indobert-lite-large-p1":  "indobenchmark-lite-large-p1",
+    "indobenchmark/indobert-lite-base-p1":   "indobenchmark-lite-base-p1",
+    "indobenchmark/indobert-large-p2":       "indobenchmark-large-p2",
+    "indobenchmark/indobert-base-p2":        "indobenchmark-base-p2",
+    "indobenchmark/indobert-lite-large-p2":  "indobenchmark-lite-large-p2",
+    "indobenchmark/indobert-lite-base-p2":   "indobenchmark-lite-base-p2",
+
+    # it
+    "dbmdz/electra-base-italian-xxl-cased-discriminator": "electra",
+
+    # ja
+    "rinna/japanese-roberta-base": "rinna-roberta",
+
+    # mr
+    "l3cube-pune/marathi-roberta": "l3cube-marathi-roberta",
+
+    # pl
+    "allegro/herbert-base-cased": "herbert",
+
+    # pt
+    "neuralmind/bert-large-portuguese-cased": "bertimbau",
+
+    # ta: tamil
+    "monsoon-nlp/tamillion":         "tamillion",
+    "lgessler/microbert-tamil-m":    "ta-microbert-m",
+    "lgessler/microbert-tamil-mxp":  "ta-microbert-mxp",
+    "l3cube-pune/tamil-bert":        "l3cube-tamil-bert",
+    "d42kw01f/Tamil-RoBERTa":        "ta-d42kw01f-roberta",
+
+    # tr
+    "dbmdz/bert-base-turkish-128k-cased": "bert",
+
+    # vi
+    "vinai/phobert-base": "phobert-base",
+    "vinai/phobert-large": "phobert-large",
+
+    # zh
+    "google-bert/bert-base-chinese": "google-bert-chinese",
+    "hfl/chinese-roberta-wwm-ext": "hfl-roberta-chinese",
+    "hfl/chinese-electra-180g-large-discriminator": "electra-large",
+
+    # multi-lingual Indic
+    "ai4bharat/indic-bert": "indic-bert",
+    "google/muril-base-cased": "muril-base-cased",
+    "google/muril-large-cased": "muril-large-cased",
+}
+
+def known_nicknames():
+    """
+    Return a list of all the transformer nicknames
+
+    We return a list so that we can sort them in decreasing key length
+    """
+    nicknames = list(value for key, value in TRANSFORMER_NICKNAMES.items())
+
+    # previously unspecific transformers get "transformer" as the nickname
+    nicknames.append("transformer")
+
+    nicknames = sorted(nicknames, key=lambda x: -len(x))
+
+    return nicknames

stanza/stanza/resources/installation.py

@@ -0,0 +1,148 @@
+"""
+Functions for setting up the environments.
+"""
+
+import os
+import logging
+import zipfile
+import shutil
+
+from stanza.resources.common import HOME_DIR, request_file, unzip, \
+    get_root_from_zipfile, set_logging_level
+
+logger = logging.getLogger('stanza')
+
+DEFAULT_CORENLP_MODEL_URL = os.getenv(
+    'CORENLP_MODEL_URL',
+    'https://huggingface.co/stanfordnlp/corenlp-{model}/resolve/{tag}/stanford-corenlp-models-{model}.jar'
+)
+BACKUP_CORENLP_MODEL_URL = "http://nlp.stanford.edu/software/stanford-corenlp-{version}-models-{model}.jar"
+
+DEFAULT_CORENLP_URL = os.getenv(
+    'CORENLP_MODEL_URL',
+    'https://huggingface.co/stanfordnlp/CoreNLP/resolve/{tag}/stanford-corenlp-latest.zip'
+)
+
+DEFAULT_CORENLP_DIR = os.getenv(
+    'CORENLP_HOME',
+    os.path.join(HOME_DIR, 'stanza_corenlp')
+)
+
+AVAILABLE_MODELS = set(['arabic', 'chinese', 'english-extra', 'english-kbp', 'french', 'german', 'hungarian', 'italian', 'spanish'])
+
+
+def download_corenlp_models(model, version, dir=DEFAULT_CORENLP_DIR, url=DEFAULT_CORENLP_MODEL_URL, logging_level='INFO', proxies=None, force=True):
+    """
+    A automatic way to download the CoreNLP models.
+
+    Args:
+        model: the name of the model, can be one of 'arabic', 'chinese', 'english',
+            'english-kbp', 'french', 'german', 'hungarian', 'italian', 'spanish'
+        version: the version of the model
+        dir: the directory to download CoreNLP model into; alternatively can be
+            set up with environment variable $CORENLP_HOME
+        url: The link to download CoreNLP models.
+             It will need {model} and either {version} or {tag} to properly format the URL
+        logging_level: logging level to use during installation
+        force: Download model anyway, no matter model file exists or not
+    """
+    dir = os.path.expanduser(dir)
+    if not model or not version:
+        raise ValueError(
+            "Both model and model version should be specified."
+        )
+    logger.info(f"Downloading {model} models (version {version}) into directory {dir}")
+    model = model.strip().lower()
+    if model not in AVAILABLE_MODELS:
+        raise KeyError(
+            f'{model} is currently not supported. '
+            f'Must be one of: {list(AVAILABLE_MODELS)}.'
+        )
+    # for example:
+    # https://huggingface.co/stanfordnlp/CoreNLP/resolve/v4.2.2/stanford-corenlp-models-french.jar
+    tag = version if version == 'main' else 'v' + version
+    download_url = url.format(tag=tag, model=model, version=version)
+    model_path = os.path.join(dir, f'stanford-corenlp-{version}-models-{model}.jar')
+
+    if os.path.exists(model_path) and not force:
+        logger.warn(
+            f"Model file {model_path} already exists. "
+            f"Please download this model to a new directory.")
+        return
+
+    try:
+        request_file(
+            download_url,
+            model_path,
+            proxies
+        )
+    except (KeyboardInterrupt, SystemExit):
+        raise
+    except Exception as e:
+        raise RuntimeError(
+            "Downloading CoreNLP model file failed. "
+            "Please try manual downloading at: https://stanfordnlp.github.io/CoreNLP/."
+        ) from e
+
+
+def install_corenlp(dir=DEFAULT_CORENLP_DIR, url=DEFAULT_CORENLP_URL, logging_level=None, proxies=None, version="main"):
+    """
+    A fully automatic way to install and setting up the CoreNLP library 
+    to use the client functionality.
+
+    Args:
+        dir: the directory to download CoreNLP model into; alternatively can be
+            set up with environment variable $CORENLP_HOME
+        url: The link to download CoreNLP models
+             Needs a {version} or {tag} parameter to specify the version
+        logging_level: logging level to use during installation
+    """
+    dir = os.path.expanduser(dir)
+    set_logging_level(logging_level=logging_level, verbose=None)
+    if os.path.exists(dir) and len(os.listdir(dir)) > 0:
+        logger.warn(
+            f"Directory {dir} already exists. "
+            f"Please install CoreNLP to a new directory.")
+        return
+
+    logger.info(f"Installing CoreNLP package into {dir}")
+    # First download the URL package
+    logger.debug(f"Download to destination file: {os.path.join(dir, 'corenlp.zip')}")
+    tag = version if version == 'main' else 'v' + version
+    url = url.format(version=version, tag=tag)
+    try:
+        request_file(url, os.path.join(dir, 'corenlp.zip'), proxies)
+
+    except (KeyboardInterrupt, SystemExit):
+        raise
+    except Exception as e:
+        raise RuntimeError(
+            "Downloading CoreNLP zip file failed. "
+            "Please try manual installation: https://stanfordnlp.github.io/CoreNLP/."
+        ) from e
+
+    # Unzip corenlp into dir
+    logger.debug("Unzipping downloaded zip file...")
+    unzip(dir, 'corenlp.zip')
+
+    # By default CoreNLP will be unzipped into a version-dependent folder, 
+    # e.g., stanford-corenlp-4.0.0. We need some hack around that and move
+    # files back into our designated folder
+    logger.debug(f"Moving files into the designated folder at: {dir}")
+    corenlp_dirname = get_root_from_zipfile(os.path.join(dir, 'corenlp.zip'))
+    corenlp_dirname = os.path.join(dir, corenlp_dirname)
+    for f in os.listdir(corenlp_dirname):
+        shutil.move(os.path.join(corenlp_dirname, f), dir)
+
+    # Remove original zip and folder
+    logger.debug("Removing downloaded zip file...")
+    os.remove(os.path.join(dir, 'corenlp.zip'))
+    shutil.rmtree(corenlp_dirname)
+
+    # Warn user to set up env
+    if dir != DEFAULT_CORENLP_DIR:
+        logger.warning(
+            f"For customized installation location, please set the `CORENLP_HOME` "
+            f"environment variable to the location of the installation. "
+            f"In Unix, this is done with `export CORENLP_HOME={dir}`.")
+

stanza/stanza/resources/prepare_resources.py

@@ -0,0 +1,670 @@
+"""
+Converts a directory of models organized by type into a directory organized by language.
+
+Also produces the resources.json file.
+
+For example, on the cluster, you can do this:
+
+python3 -m stanza.resources.prepare_resources --input_dir /u/nlp/software/stanza/models/current-models-1.5.0 --output_dir /u/nlp/software/stanza/models/1.5.0 > resources.out 2>&1
+nlprun -a stanza-1.2 -q john "python3 -m stanza.resources.prepare_resources --input_dir /u/nlp/software/stanza/models/current-models-1.5.0 --output_dir /u/nlp/software/stanza/models/1.5.0" -o resources.out
+"""
+
+import argparse
+from collections import defaultdict
+import json
+import os
+from pathlib import Path
+import hashlib
+import shutil
+import zipfile
+
+from stanza import __resources_version__
+from stanza.models.common.constant import lcode2lang, two_to_three_letters, three_to_two_letters
+from stanza.resources.default_packages import PACKAGES, TRANSFORMERS, TRANSFORMER_NICKNAMES
+from stanza.resources.default_packages import *
+from stanza.utils.datasets.prepare_lemma_classifier import DATASET_MAPPING as LEMMA_CLASSIFIER_DATASETS
+from stanza.utils.get_tqdm import get_tqdm
+
+tqdm = get_tqdm()
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--input_dir', type=str, default="/u/nlp/software/stanza/models/current-models-%s" % __resources_version__, help='Input dir for various models.  Defaults to the recommended home on the nlp cluster')
+    parser.add_argument('--output_dir', type=str, default="/u/nlp/software/stanza/models/%s" % __resources_version__, help='Output dir for various models.')
+    parser.add_argument('--packages_only', action='store_true', default=False, help='Only build the package maps instead of rebuilding everything')
+    parser.add_argument('--lang', type=str, default=None, help='Only process this language or a comma-separated list of languages.  If left blank, will prepare all languages.  To use this argument, a previous prepared resources with all of the languages is necessary.')
+    args = parser.parse_args()
+    args.input_dir = os.path.abspath(args.input_dir)
+    args.output_dir = os.path.abspath(args.output_dir)
+    if args.lang is not None:
+        args.lang = ",".join(args.lang.strip().split())
+    return args
+
+
+allowed_empty_languages = [
+    # we don't have a lot of Thai support yet
+    "th",
+    # only tokenize and NER for Myanmar right now (soon...)
+    "my",
+]
+
+# map processor name to file ending
+# the order of this dict determines the order in which default.zip files are built
+# changing it will necessitate rebuilding all of the default.zip files
+# not a disaster, but it would involve a bunch of uploading
+processor_to_ending = {
+    "tokenize": "tokenizer",
+    "mwt": "mwt_expander",
+    "lemma": "lemmatizer",
+    "pos": "tagger",
+    "depparse": "parser",
+    "pretrain": "pretrain",
+    "ner": "nertagger",
+    "forward_charlm": "forward_charlm",
+    "backward_charlm": "backward_charlm",
+    "sentiment": "sentiment",
+    "constituency": "constituency",
+    "coref": "coref",
+    "langid": "langid",
+}
+ending_to_processor = {j: i for i, j in processor_to_ending.items()}
+PROCESSORS = list(processor_to_ending.keys())
+
+def ensure_dir(dir):
+    Path(dir).mkdir(parents=True, exist_ok=True)
+
+
+def copy_file(src, dst):
+    ensure_dir(Path(dst).parent)
+    shutil.copy2(src, dst)
+
+
+def get_md5(path):
+    data = open(path, 'rb').read()
+    return hashlib.md5(data).hexdigest()
+
+
+def split_model_name(model):
+    """
+    Split model names by _
+
+    Takes into account packages with _ and processor types with _
+    """
+    model = model[:-3].replace('.', '_')
+    # sort by key length so that nertagger is checked before tagger, for example
+    for processor in sorted(ending_to_processor.keys(), key=lambda x: -len(x)):
+        if model.endswith(processor):
+            model = model[:-(len(processor)+1)]
+            processor = ending_to_processor[processor]
+            break
+    else:
+        raise AssertionError(f"Could not find a processor type in {model}")
+    lang, package = model.split('_', 1)
+    return lang, package, processor
+
+def split_package(package):
+    if package.endswith("_finetuned"):
+        package = package[:-10]
+
+    if package.endswith("_nopretrain"):
+        package = package[:-11]
+        return package, False, False
+    if package.endswith("_nocharlm"):
+        package = package[:-9]
+        return package, True, False
+    if package.endswith("_charlm"):
+        package = package[:-7]
+        return package, True, True
+    underscore = package.rfind("_")
+    if underscore >= 0:
+        # +1 to skip the underscore
+        nickname = package[underscore+1:]
+        if nickname in known_nicknames():
+            return package[:underscore], True, True
+
+    # guess it was a model which wasn't built with the new naming convention of putting the pretrain type at the end
+    # assume WV and charlm... if the language / package doesn't allow for one, that should be caught later
+    return package, True, True
+
+def get_pretrain_package(lang, package, model_pretrains, default_pretrains):
+    package, uses_pretrain, _ = split_package(package)
+
+    if not uses_pretrain or lang in no_pretrain_languages:
+        return None
+    elif model_pretrains is not None and lang in model_pretrains and package in model_pretrains[lang]:
+        return model_pretrains[lang][package]
+    elif lang in default_pretrains:
+        return default_pretrains[lang]
+
+    raise RuntimeError("pretrain not specified for lang %s package %s" % (lang, package))
+
+def get_charlm_package(lang, package, model_charlms, default_charlms):
+    package, _, uses_charlm = split_package(package)
+
+    if not uses_charlm:
+        return None
+
+    if model_charlms is not None and lang in model_charlms and package in model_charlms[lang]:
+        return model_charlms[lang][package]
+    else:
+        return default_charlms.get(lang, None)
+
+def get_con_dependencies(lang, package):
+    # so far, this invariant is true:
+    # constituency models use the default pretrain and charlm for the language
+    # sometimes there is no charlm for a language that has constituency, though
+    pretrain_package = get_pretrain_package(lang, package, None, default_pretrains)
+    dependencies = [{'model': 'pretrain', 'package': pretrain_package}]
+
+    charlm_package = default_charlms.get(lang, None)
+    if charlm_package is not None:
+        dependencies.append({'model': 'forward_charlm', 'package': charlm_package})
+        dependencies.append({'model': 'backward_charlm', 'package': charlm_package})
+
+    return dependencies
+
+def get_pos_charlm_package(lang, package):
+    return get_charlm_package(lang, package, pos_charlms, default_charlms)
+
+def get_pos_dependencies(lang, package):
+    dependencies = []
+
+    pretrain_package = get_pretrain_package(lang, package, pos_pretrains, default_pretrains)
+    if pretrain_package is not None:
+        dependencies.append({'model': 'pretrain', 'package': pretrain_package})
+
+    charlm_package = get_pos_charlm_package(lang, package)
+    if charlm_package is not None:
+        dependencies.append({'model': 'forward_charlm', 'package': charlm_package})
+        dependencies.append({'model': 'backward_charlm', 'package': charlm_package})
+
+    return dependencies
+
+def get_lemma_pretrain_package(lang, package):
+    package, uses_pretrain, uses_charlm = split_package(package)
+    if not uses_pretrain:
+        return None
+    if not uses_charlm:
+        # currently the contextual lemma classifier is only active
+        # for the charlm lemmatizers
+        return None
+    if "%s_%s" % (lang, package) not in LEMMA_CLASSIFIER_DATASETS:
+        return None
+    return get_pretrain_package(lang, package, {}, default_pretrains)
+
+def get_lemma_charlm_package(lang, package):
+    return get_charlm_package(lang, package, lemma_charlms, default_charlms)
+
+def get_lemma_dependencies(lang, package):
+    dependencies = []
+
+    pretrain_package = get_lemma_pretrain_package(lang, package)
+    if pretrain_package is not None:
+        dependencies.append({'model': 'pretrain', 'package': pretrain_package})
+
+    charlm_package = get_lemma_charlm_package(lang, package)
+    if charlm_package is not None:
+        dependencies.append({'model': 'forward_charlm', 'package': charlm_package})
+        dependencies.append({'model': 'backward_charlm', 'package': charlm_package})
+
+    return dependencies
+
+
+def get_depparse_charlm_package(lang, package):
+    return get_charlm_package(lang, package, depparse_charlms, default_charlms)
+
+def get_depparse_dependencies(lang, package):
+    dependencies = []
+
+    pretrain_package = get_pretrain_package(lang, package, depparse_pretrains, default_pretrains)
+    if pretrain_package is not None:
+        dependencies.append({'model': 'pretrain', 'package': pretrain_package})
+
+    charlm_package = get_depparse_charlm_package(lang, package)
+    if charlm_package is not None:
+        dependencies.append({'model': 'forward_charlm', 'package': charlm_package})
+        dependencies.append({'model': 'backward_charlm', 'package': charlm_package})
+
+    return dependencies
+
+def get_ner_charlm_package(lang, package):
+    return get_charlm_package(lang, package, ner_charlms, default_charlms)
+
+def get_ner_pretrain_package(lang, package):
+    return get_pretrain_package(lang, package, ner_pretrains, default_pretrains)
+
+def get_ner_dependencies(lang, package):
+    dependencies = []
+
+    pretrain_package = get_ner_pretrain_package(lang, package)
+    if pretrain_package is not None:
+        dependencies.append({'model': 'pretrain', 'package': pretrain_package})
+
+    charlm_package = get_ner_charlm_package(lang, package)
+    if charlm_package is not None:
+        dependencies.append({'model': 'forward_charlm', 'package': charlm_package})
+        dependencies.append({'model': 'backward_charlm', 'package': charlm_package})
+
+    return dependencies
+
+def get_sentiment_dependencies(lang, package):
+    """
+    Return a list of dependencies for the sentiment model
+
+    Generally this will be pretrain, forward & backward charlm
+    So far, this invariant is true:
+    sentiment models use the default pretrain for the language
+    also, they all use the default charlm for a language
+    """
+    pretrain_package = get_pretrain_package(lang, package, None, default_pretrains)
+    dependencies = [{'model': 'pretrain', 'package': pretrain_package}]
+
+    charlm_package = default_charlms.get(lang, None)
+    if charlm_package is not None:
+        dependencies.append({'model': 'forward_charlm', 'package': charlm_package})
+        dependencies.append({'model': 'backward_charlm', 'package': charlm_package})
+
+    return dependencies
+
+def get_dependencies(processor, lang, package):
+    """
+    Get the dependencies for a particular lang/package based on the package name
+
+    The package can include descriptors such as _nopretrain, _nocharlm, _charlm
+    which inform whether or not this particular model uses charlm or pretrain
+    """
+    if processor == 'depparse':
+        return get_depparse_dependencies(lang, package)
+    elif processor == 'lemma':
+        return get_lemma_dependencies(lang, package)
+    elif processor == 'pos':
+        return get_pos_dependencies(lang, package)
+    elif processor == 'ner':
+        return get_ner_dependencies(lang, package)
+    elif processor == 'sentiment':
+        return get_sentiment_dependencies(lang, package)
+    elif processor == 'constituency':
+        return get_con_dependencies(lang, package)
+    return {}
+
+def process_dirs(args):
+    dirs = sorted(os.listdir(args.input_dir))
+    resources = {}
+    if args.lang:
+        resources = json.load(open(os.path.join(args.output_dir, 'resources.json')))
+        # this one language gets overridden
+        # if this is not done, and we reuse the old resources,
+        # any models which were deleted will still be in the resources
+        for lang in args.lang.split(","):
+            resources[lang] = {}
+
+    for model_dir in dirs:
+        print(f"Processing models in {model_dir}")
+        models = sorted(os.listdir(os.path.join(args.input_dir, model_dir)))
+        for model in tqdm(models):
+            if not model.endswith('.pt'): continue
+            # get processor
+            lang, package, processor = split_model_name(model)
+            if args.lang and lang not in args.lang.split(","):
+                continue
+
+            # copy file
+            input_path = os.path.join(args.input_dir, model_dir, model)
+            output_path = os.path.join(args.output_dir, lang, "models", processor, package + '.pt')
+            copy_file(input_path, output_path)
+            # maintain md5
+            md5 = get_md5(output_path)
+            # maintain dependencies
+            dependencies = get_dependencies(processor, lang, package)
+            # maintain resources
+            if lang not in resources: resources[lang] = {}
+            if processor not in resources[lang]: resources[lang][processor] = {}
+            if dependencies:
+                resources[lang][processor][package] = {'md5': md5, 'dependencies': dependencies}
+            else:
+                resources[lang][processor][package] = {'md5': md5}
+    print("Processed initial model directories.  Writing preliminary resources.json")
+    json.dump(resources, open(os.path.join(args.output_dir, 'resources.json'), 'w'), indent=2)
+
+def get_default_pos_package(lang, ud_package):
+    charlm_package = get_pos_charlm_package(lang, ud_package)
+    if charlm_package is not None:
+        return ud_package + "_charlm"
+    if lang in no_pretrain_languages:
+        return ud_package + "_nopretrain"
+    return ud_package + "_nocharlm"
+
+def get_default_depparse_package(lang, ud_package):
+    charlm_package = get_depparse_charlm_package(lang, ud_package)
+    if charlm_package is not None:
+        return ud_package + "_charlm"
+    if lang in no_pretrain_languages:
+        return ud_package + "_nopretrain"
+    return ud_package + "_nocharlm"
+
+def process_default_zips(args):
+    resources = json.load(open(os.path.join(args.output_dir, 'resources.json')))
+    for lang in resources:
+        # check url, alias, and lang_name in case we are rerunning this step on an already built resources.json
+        if lang == 'url':
+            continue
+        if 'alias' in resources[lang]:
+            continue
+        if all(k in ("backward_charlm", "forward_charlm", "pretrain", "lang_name") for k in resources[lang].keys()):
+            continue
+        if lang not in default_treebanks:
+            raise AssertionError(f'{lang} not in default treebanks!!!')
+
+        if args.lang and lang not in args.lang.split(","):
+            continue
+
+        print(f'Preparing default models for language {lang}')
+
+        models_needed = defaultdict(set)
+
+        packages = resources[lang][PACKAGES]["default"]
+        for processor, package in packages.items():
+            if processor == 'lemma' and package == 'identity':
+                continue
+            if processor == 'optional':
+                continue
+            models_needed[processor].add(package)
+            dependencies = get_dependencies(processor, lang, package)
+            for dependency in dependencies:
+                models_needed[dependency['model']].add(dependency['package'])
+
+        model_files = []
+        for processor in PROCESSORS:
+            if processor in models_needed:
+                for package in sorted(models_needed[processor]):
+                    filename = os.path.join(args.output_dir, lang, "models", processor, package + '.pt')
+                    if os.path.exists(filename):
+                        print("   Model {} package {}: file {}".format(processor, package, filename))
+                        model_files.append((filename, processor, package))
+                    else:
+                        raise FileNotFoundError(f"Processor {processor} package {package} needed for {lang} but cannot be found at {filename}")
+
+        with zipfile.ZipFile(os.path.join(args.output_dir, lang, 'models', 'default.zip'), 'w', zipfile.ZIP_DEFLATED) as zipf:
+            for filename, processor, package in model_files:
+                zipf.write(filename=filename, arcname=os.path.join(processor, package + '.pt'))
+
+        default_md5 = get_md5(os.path.join(args.output_dir, lang, 'models', 'default.zip'))
+        resources[lang]['default_md5'] = default_md5
+
+    print("Processed default model zips.  Writing resources.json")
+    json.dump(resources, open(os.path.join(args.output_dir, 'resources.json'), 'w'), indent=2)
+
+def get_default_processors(resources, lang):
+    """
+    Build a default package for this language
+
+    Will add each of pos, lemma, depparse, etc if those are available
+    Uses the existing models scraped from the language directories into resources.json, as relevant
+    """
+    if lang == "multilingual":
+        return {"langid": "ud"}
+
+    default_package = default_treebanks[lang]
+    default_processors = {}
+    if lang in default_tokenizer:
+        default_processors['tokenize'] = default_tokenizer[lang]
+    else:
+        default_processors['tokenize'] = default_package
+
+    if 'mwt' in resources[lang] and default_processors['tokenize'] in resources[lang]['mwt']:
+        # if this doesn't happen, we just skip MWT
+        default_processors['mwt'] = default_package
+
+    if 'lemma' in resources[lang]:
+        expected_lemma = default_package + "_nocharlm"
+        if expected_lemma in resources[lang]['lemma']:
+            default_processors['lemma'] = expected_lemma
+    elif lang not in allowed_empty_languages:
+        default_processors['lemma'] = 'identity'
+
+    if 'pos' in resources[lang]:
+        default_processors['pos'] = get_default_pos_package(lang, default_package)
+        if default_processors['pos'] not in resources[lang]['pos']:
+            raise AssertionError("Expected POS model not in resources: %s" % default_processors['pos'])
+    elif lang not in allowed_empty_languages:
+        raise AssertionError("Expected to find POS models for language %s" % lang)
+
+    if 'depparse' in resources[lang]:
+        default_processors['depparse'] = get_default_depparse_package(lang, default_package)
+        if default_processors['depparse'] not in resources[lang]['depparse']:
+            raise AssertionError("Expected depparse model not in resources: %s" % default_processors['depparse'])
+    elif lang not in allowed_empty_languages:
+        raise AssertionError("Expected to find depparse models for language %s" % lang)
+
+    if lang in default_ners:
+        default_processors['ner'] = default_ners[lang]
+
+    if lang in default_sentiment:
+        default_processors['sentiment'] = default_sentiment[lang]
+
+    if lang in default_constituency:
+        default_processors['constituency'] = default_constituency[lang]
+
+    optional = get_default_optional_processors(resources, lang)
+    if optional:
+        default_processors['optional'] = optional
+
+    return default_processors
+
+def get_default_optional_processors(resources, lang):
+    optional_processors = {}
+    if lang in optional_constituency:
+        optional_processors['constituency'] = optional_constituency[lang]
+
+    if lang in optional_coref:
+        optional_processors['coref'] = optional_coref[lang]
+
+    return optional_processors
+
+def update_processor_add_transformer(resources, lang, current_processors, processor, transformer):
+    if processor not in current_processors:
+        return
+
+    new_model = current_processors[processor].replace('_charlm', "_" + transformer).replace('_nocharlm', "_" + transformer)
+    if new_model in resources[lang][processor]:
+        current_processors[processor] = new_model
+    else:
+        print("WARNING: wanted to use %s for %s accurate %s, but that model does not exist" % (new_model, lang, processor))
+
+def get_default_accurate(resources, lang):
+    """
+    A package that, if available, uses charlm and transformer models for each processor
+    """
+    default_processors = get_default_processors(resources, lang)
+
+    if 'lemma' in default_processors and default_processors['lemma'] != 'identity':
+        lemma_model = default_processors['lemma']
+        lemma_model = lemma_model.replace('_nocharlm', '_charlm')
+        charlm_package = get_lemma_charlm_package(lang, lemma_model)
+        if charlm_package is not None:
+            if lemma_model in resources[lang]['lemma']:
+                default_processors['lemma'] = lemma_model
+            else:
+                print("WARNING: wanted to use %s for %s default_accurate lemma, but that model does not exist" % (lemma_model, lang))
+
+    transformer = TRANSFORMER_NICKNAMES.get(TRANSFORMERS.get(lang, None), None)
+    if transformer is not None:
+        for processor in ('pos', 'depparse', 'constituency', 'sentiment'):
+            update_processor_add_transformer(resources, lang, default_processors, processor, transformer)
+        if 'ner' in default_processors and (default_processors['ner'].endswith("_charlm") or default_processors['ner'].endswith("_nocharlm")):
+            update_processor_add_transformer(resources, lang, default_processors, "ner", transformer)
+
+    optional = get_optional_accurate(resources, lang)
+    if optional:
+        default_processors['optional'] = optional
+
+    return default_processors
+
+def get_optional_accurate(resources, lang):
+    optional_processors = get_default_optional_processors(resources, lang)
+
+    transformer = TRANSFORMER_NICKNAMES.get(TRANSFORMERS.get(lang, None), None)
+    if transformer is not None:
+        for processor in ('pos', 'depparse', 'constituency', 'sentiment'):
+            update_processor_add_transformer(resources, lang, optional_processors, processor, transformer)
+
+    if lang in optional_coref:
+        optional_processors['coref'] = optional_coref[lang]
+
+    return optional_processors
+
+
+def get_default_fast(resources, lang):
+    """
+    Build a packages entry which only has the nocharlm models
+
+    Will make it easy for people to use the lower tier of models
+
+    We do this by building the same default package as normal,
+    then switching everything out for the lower tier model when possible.
+    We also remove constituency, as it is super slow.
+    Note that in the case of a language which doesn't have a charlm,
+    that means we wind up building the same for default and default_nocharlm
+    """
+    default_processors = get_default_processors(resources, lang)
+
+    # this is a slow model and we don't have non-charlm versions of it yet
+    if 'constituency' in default_processors:
+        default_processors.pop('constituency')
+
+    for processor, model in default_processors.items():
+        if "_charlm" in model:
+            nocharlm = model.replace("_charlm", "_nocharlm")
+            if nocharlm not in resources[lang][processor]:
+                print("WARNING: wanted to use %s for %s default_fast processor %s, but that model does not exist" % (nocharlm, lang, processor))
+            else:
+                default_processors[processor] = nocharlm
+
+    return default_processors
+
+def process_packages(args):
+    """
+    Build a package for a language's default processors and all of the treebanks specifically used for that language
+    """
+    resources = json.load(open(os.path.join(args.output_dir, 'resources.json')))
+
+    for lang in resources:
+        # check url, alias, and lang_name in case we are rerunning this step on an already built resources.json
+        if lang == 'url':
+            continue
+        if 'alias' in resources[lang]:
+            continue
+        if all(k in ("backward_charlm", "forward_charlm", "pretrain", "lang_name") for k in resources[lang].keys()):
+            continue
+        if lang not in default_treebanks:
+            raise AssertionError(f'{lang} not in default treebanks!!!')
+
+        if args.lang and lang not in args.lang.split(","):
+            continue
+
+        default_processors = get_default_processors(resources, lang)
+
+        # TODO: eventually we can remove default_processors
+        # For now, we want to keep this so that v1.5.1 is compatible
+        # with the next iteration of resources files
+        resources[lang]['default_processors'] = default_processors
+        resources[lang][PACKAGES] = {}
+        resources[lang][PACKAGES]['default'] = default_processors
+
+        if lang not in no_pretrain_languages and lang != "multilingual":
+            default_fast = get_default_fast(resources, lang)
+            resources[lang][PACKAGES]['default_fast'] = default_fast
+
+            default_accurate = get_default_accurate(resources, lang)
+            resources[lang][PACKAGES]['default_accurate'] = default_accurate
+
+        # Now we loop over each of the tokenizers for this language
+        # ... we use this as a proxy for the available UD treebanks
+        # This loop also catches things such as "craft" which are
+        # included treebanks that aren't UD
+        # We then create a package in the packages dict for each of those treebanks
+        if 'tokenize' in resources[lang]:
+            for package in resources[lang]['tokenize']:
+                processors = {"tokenize": package}
+                if "mwt" in resources[lang] and package in resources[lang]["mwt"]:
+                    processors["mwt"] = package
+
+                if "pos" in resources[lang]:
+                    if package + "_charlm" in resources[lang]["pos"]:
+                        processors["pos"] = package + "_charlm"
+                    elif package + "_nocharlm" in resources[lang]["pos"]:
+                        processors["pos"] = package + "_nocharlm"
+
+                if "lemma" in resources[lang] and "pos" in processors:
+                    lemma_package = package + "_nocharlm"
+                    if lemma_package in resources[lang]["lemma"]:
+                        processors["lemma"] = lemma_package
+
+                if "depparse" in resources[lang] and "pos" in processors:
+                    depparse_package = None
+                    if package + "_charlm" in resources[lang]["depparse"]:
+                        depparse_package = package + "_charlm"
+                    elif package + "_nocharlm" in resources[lang]["depparse"]:
+                        depparse_package = package + "_nocharlm"
+                    # we want to set the lemma first if it's identity
+                    # THEN set the depparse
+                    if depparse_package is not None:
+                        if "lemma" not in processors:
+                            processors["lemma"] = "identity"
+                        processors["depparse"] = depparse_package
+
+                resources[lang][PACKAGES][package] = processors
+
+    print("Processed packages.  Writing resources.json")
+    json.dump(resources, open(os.path.join(args.output_dir, 'resources.json'), 'w'), indent=2)
+
+def process_lcode(args):
+    resources = json.load(open(os.path.join(args.output_dir, 'resources.json')))
+    resources_new = {}
+    resources_new["multilingual"] = resources["multilingual"]
+    for lang in resources:
+        if lang == 'multilingual':
+            continue
+        if 'alias' in resources[lang]:
+            continue
+        if lang not in lcode2lang:
+            print(lang + ' not found in lcode2lang!')
+            continue
+        lang_name = lcode2lang[lang]
+        resources[lang]['lang_name'] = lang_name
+        resources_new[lang.lower()] = resources[lang.lower()]
+        resources_new[lang_name.lower()] = {'alias': lang.lower()}
+        if lang.lower() in two_to_three_letters:
+            resources_new[two_to_three_letters[lang.lower()]] = {'alias': lang.lower()}
+        elif lang.lower() in three_to_two_letters:
+            resources_new[three_to_two_letters[lang.lower()]] = {'alias': lang.lower()}
+    print("Processed lcode aliases.  Writing resources.json")
+    json.dump(resources_new, open(os.path.join(args.output_dir, 'resources.json'), 'w'), indent=2)
+
+
+def process_misc(args):
+    resources = json.load(open(os.path.join(args.output_dir, 'resources.json')))
+    resources['no'] = {'alias': 'nb'}
+    resources['zh'] = {'alias': 'zh-hans'}
+    # This is intended to be unformatted.  expand_model_url in common.py will fill in the raw string
+    # with the appropriate values in order to find the needed model file on huggingface
+    resources['url'] = 'https://huggingface.co/stanfordnlp/stanza-{lang}/resolve/v{resources_version}/models/{filename}'
+    print("Finalized misc attributes.  Writing resources.json")
+    json.dump(resources, open(os.path.join(args.output_dir, 'resources.json'), 'w'), indent=2)
+
+
+def main():
+    args = parse_args()
+    print("Converting models from %s to %s" % (args.input_dir, args.output_dir))
+    if not args.packages_only:
+        process_dirs(args)
+    process_packages(args)
+    if not args.packages_only:
+        process_default_zips(args)
+        process_lcode(args)
+        process_misc(args)
+
+
+if __name__ == '__main__':
+    main()
+

stanza/stanza/server/__init__.py

@@ -0,0 +1,10 @@
+from stanza.protobuf import to_text
+from stanza.protobuf import Document, Sentence, Token, IndexedWord, Span
+from stanza.protobuf import ParseTree, DependencyGraph, CorefChain
+from stanza.protobuf import Mention, NERMention, Entity, Relation, RelationTriple, Timex
+from stanza.protobuf import Quote, SpeakerInfo
+from stanza.protobuf import Operator, Polarity
+from stanza.protobuf import SentenceFragment, TokenLocation
+from stanza.protobuf import MapStringString, MapIntString
+from .client import CoreNLPClient, AnnotationException, TimeoutException, PermanentlyFailedException, StartServer
+from .annotator import Annotator

stanza/stanza/server/annotator.py

@@ -0,0 +1,138 @@
+"""
+Defines a base class that can be used to annotate.
+"""
+import io
+from multiprocessing import Process
+from http.server import BaseHTTPRequestHandler, HTTPServer
+from http import client as HTTPStatus
+
+from stanza.protobuf import Document, parseFromDelimitedString, writeToDelimitedString
+
+class Annotator(Process):
+    """
+    This annotator base class hosts a lightweight server that accepts
+    annotation requests from CoreNLP.
+    Each annotator simply defines 3 functions: requires, provides and annotate.
+
+    This class takes care of defining appropriate endpoints to interface
+    with CoreNLP.
+    """
+    @property
+    def name(self):
+        """
+        Name of the annotator (used by CoreNLP)
+        """
+        raise NotImplementedError()
+
+    @property
+    def requires(self):
+        """
+        Requires has to specify all the annotations required before we
+        are called.
+        """
+        raise NotImplementedError()
+
+    @property
+    def provides(self):
+        """
+        The set of annotations guaranteed to be provided when we are done.
+        NOTE: that these annotations are either fully qualified Java
+        class names or refer to nested classes of
+        edu.stanford.nlp.ling.CoreAnnotations (as is the case below).
+        """
+        raise NotImplementedError()
+
+    def annotate(self, ann):
+        """
+        @ann: is a protobuf annotation object.
+        Actually populate @ann with tokens.
+        """
+        raise NotImplementedError()
+
+    @property
+    def properties(self):
+        """
+        Defines a Java property to define this annotator to CoreNLP.
+        """
+        return {
+            "customAnnotatorClass.{}".format(self.name): "edu.stanford.nlp.pipeline.GenericWebServiceAnnotator",
+            "generic.endpoint": "http://{}:{}".format(self.host, self.port),
+            "generic.requires": ",".join(self.requires),
+            "generic.provides": ",".join(self.provides),
+            }
+
+    class _Handler(BaseHTTPRequestHandler):
+        annotator = None
+
+        def __init__(self, request, client_address, server):
+            BaseHTTPRequestHandler.__init__(self, request, client_address, server)
+
+        def do_GET(self):
+            """
+            Handle a ping request
+            """
+            if not self.path.endswith("/"): self.path += "/"
+            if self.path == "/ping/":
+                msg = "pong".encode("UTF-8")
+
+                self.send_response(HTTPStatus.OK)
+                self.send_header("Content-Type", "text/application")
+                self.send_header("Content-Length", len(msg))
+                self.end_headers()
+                self.wfile.write(msg)
+            else:
+                self.send_response(HTTPStatus.BAD_REQUEST)
+                self.end_headers()
+
+        def do_POST(self):
+            """
+            Handle an annotate request
+            """
+            if not self.path.endswith("/"): self.path += "/"
+            if self.path == "/annotate/":
+                # Read message
+                length = int(self.headers.get('content-length'))
+                msg = self.rfile.read(length)
+
+                # Do the annotation
+                doc = Document()
+                parseFromDelimitedString(doc, msg)
+                self.annotator.annotate(doc)
+
+                with io.BytesIO() as stream:
+                    writeToDelimitedString(doc, stream)
+                    msg = stream.getvalue()
+
+                # write message
+                self.send_response(HTTPStatus.OK)
+                self.send_header("Content-Type", "application/x-protobuf")
+                self.send_header("Content-Length", len(msg))
+                self.end_headers()
+                self.wfile.write(msg)
+
+            else:
+                self.send_response(HTTPStatus.BAD_REQUEST)
+                self.end_headers()
+
+    def __init__(self, host="", port=8432):
+        """
+        Launches a server endpoint to communicate with CoreNLP
+        """
+        Process.__init__(self)
+        self.host, self.port = host, port
+        self._Handler.annotator = self
+
+    def run(self):
+        """
+        Runs the server using Python's simple HTTPServer.
+        TODO: make this multithreaded.
+        """
+        httpd = HTTPServer((self.host, self.port), self._Handler)
+        sa = httpd.socket.getsockname()
+        serve_message = "Serving HTTP on {host} port {port} (http://{host}:{port}/) ..."
+        print(serve_message.format(host=sa[0], port=sa[1]))
+        try:
+            httpd.serve_forever()
+        except KeyboardInterrupt:
+            print("\nKeyboard interrupt received, exiting.")
+            httpd.shutdown()

stanza/stanza/server/client.py

@@ -0,0 +1,779 @@
+"""
+Client for accessing Stanford CoreNLP in Python
+"""
+
+import atexit
+import contextlib
+import enum
+import io
+import os
+import re
+import requests
+import logging
+import json
+import shlex
+import socket
+import subprocess
+import time
+import sys
+import uuid
+
+from datetime import datetime
+from pathlib import Path
+from urllib.parse import urlparse
+
+from stanza.protobuf import Document, parseFromDelimitedString, writeToDelimitedString, to_text
+__author__ = 'arunchaganty, kelvinguu, vzhong, wmonroe4'
+
+logger = logging.getLogger('stanza')
+
+# pattern tmp props file should follow
+SERVER_PROPS_TMP_FILE_PATTERN = re.compile('corenlp_server-(.*).props')
+
+# Check if str is CoreNLP supported language
+CORENLP_LANGS = ['ar', 'arabic', 'chinese', 'zh', 'english', 'en', 'french', 'fr', 'de', 'german', 'hu', 'hungarian',
+                 'it', 'italian', 'es', 'spanish']
+
+# map shorthands to full language names
+LANGUAGE_SHORTHANDS_TO_FULL = {
+    "ar": "arabic",
+    "zh": "chinese",
+    "en": "english",
+    "fr": "french",
+    "de": "german",
+    "hu": "hungarian",
+    "it": "italian",
+    "es": "spanish"
+}
+
+
+def is_corenlp_lang(props_str):
+    """ Check if a string references a CoreNLP language """
+    return props_str.lower() in CORENLP_LANGS
+
+
+# Validate CoreNLP properties
+CORENLP_OUTPUT_VALS = ["conll", "conllu", "json", "serialized", "text", "xml", "inlinexml"]
+
+
+def validate_corenlp_props(properties=None, annotators=None, output_format=None):
+    """ Do basic checks to validate CoreNLP properties """
+    if output_format and output_format.lower() not in CORENLP_OUTPUT_VALS:
+        raise ValueError(f"{output_format} not a valid CoreNLP outputFormat value! Choose from: {CORENLP_OUTPUT_VALS}")
+    if type(properties) == dict:
+        if "outputFormat" in properties and properties["outputFormat"].lower() not in CORENLP_OUTPUT_VALS:
+            raise ValueError(f"{properties['outputFormat']} not a valid CoreNLP outputFormat value! Choose from: "
+                             f"{CORENLP_OUTPUT_VALS}")
+
+
+class AnnotationException(Exception):
+    """ Exception raised when there was an error communicating with the CoreNLP server. """
+    pass
+
+
+class TimeoutException(AnnotationException):
+    """ Exception raised when the CoreNLP server timed out. """
+    pass
+
+
+class ShouldRetryException(Exception):
+    """ Exception raised if the service should retry the request. """
+    pass
+
+
+class PermanentlyFailedException(Exception):
+    """ Exception raised if the service should NOT retry the request. """
+    pass
+
+class StartServer(enum.Enum):
+    DONT_START = 0
+    FORCE_START = 1
+    TRY_START = 2
+
+
+def clean_props_file(props_file):
+    # check if there is a temp server props file to remove and remove it
+    if props_file:
+        if os.path.isfile(props_file) and SERVER_PROPS_TMP_FILE_PATTERN.match(os.path.basename(props_file)):
+            os.remove(props_file)
+
+
+class RobustService(object):
+    """ Service that resuscitates itself if it is not available. """
+    CHECK_ALIVE_TIMEOUT = 120
+
+    def __init__(self, start_cmd, stop_cmd, endpoint, stdout=None,
+                 stderr=None, be_quiet=False, host=None, port=None, ignore_binding_error=False):
+        self.start_cmd = start_cmd and shlex.split(start_cmd)
+        self.stop_cmd = stop_cmd and shlex.split(stop_cmd)
+        self.endpoint = endpoint
+        self.stdout = stdout
+        self.stderr = stderr
+
+        self.server = None
+        self.is_active = False
+        self.be_quiet = be_quiet
+        self.host = host
+        self.port = port
+        self.ignore_binding_error = ignore_binding_error
+        atexit.register(self.atexit_kill)
+
+    def is_alive(self):
+        try:
+            if not self.ignore_binding_error and self.server is not None and self.server.poll() is not None:
+                return False
+            return requests.get(self.endpoint + "/ping").ok
+        except requests.exceptions.ConnectionError as e:
+            raise ShouldRetryException(e)
+
+    def start(self):
+        if self.start_cmd:
+            if self.host and self.port:
+                with contextlib.closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock:
+                    try:
+                        sock.bind((self.host, self.port))
+                    except socket.error as e:
+                        if self.ignore_binding_error:
+                            logger.info(f"Connecting to existing CoreNLP server at {self.host}:{self.port}")
+                            self.server = None
+                            return
+                        else:
+                            raise PermanentlyFailedException("Error: unable to start the CoreNLP server on port %d "
+                                                             "(possibly something is already running there)" % self.port) from e
+            if self.be_quiet:
+                # Issue #26: subprocess.DEVNULL isn't supported in python 2.7.
+                if hasattr(subprocess, 'DEVNULL'):
+                    stderr = subprocess.DEVNULL
+                else:
+                    stderr = open(os.devnull, 'w')
+                stdout = stderr
+            else:
+                stdout = self.stdout
+                stderr = self.stderr
+            logger.info(f"Starting server with command: {' '.join(self.start_cmd)}")
+            try:
+                self.server = subprocess.Popen(self.start_cmd,
+                                               stderr=stderr,
+                                               stdout=stdout)
+            except FileNotFoundError as e:
+                raise FileNotFoundError("When trying to run CoreNLP, a FileNotFoundError occurred, which frequently means Java was not installed or was not in the classpath.") from e
+
+    def atexit_kill(self):
+        # make some kind of effort to stop the service (such as a
+        # CoreNLP server) at the end of the program.  not waiting so
+        # that the python script exiting isn't delayed
+        if self.server and self.server.poll() is None:
+            self.server.terminate()
+
+    def stop(self):
+        if self.server:
+            self.server.terminate()
+            try:
+                self.server.wait(5)
+            except subprocess.TimeoutExpired:
+                # Resorting to more aggressive measures...
+                self.server.kill()
+                try:
+                    self.server.wait(5)
+                except subprocess.TimeoutExpired:
+                    # oh well
+                    pass
+            self.server = None
+        if self.stop_cmd:
+            subprocess.run(self.stop_cmd, check=True)
+        self.is_active = False
+
+    def __enter__(self):
+        self.start()
+        return self
+
+    def __exit__(self, _, __, ___):
+        self.stop()
+
+    def ensure_alive(self):
+        # Check if the service is active and alive
+        if self.is_active:
+            try:
+                if self.is_alive():
+                    return
+                else:
+                    self.stop()
+            except ShouldRetryException:
+                pass
+
+        # If not, try to start up the service.
+        if self.server is None:
+            self.start()
+
+        # Wait for the service to start up.
+        start_time = time.time()
+        while True:
+            try:
+                if self.is_alive():
+                    break
+            except ShouldRetryException:
+                pass
+
+            if time.time() - start_time < self.CHECK_ALIVE_TIMEOUT:
+                time.sleep(1)
+            else:
+                raise PermanentlyFailedException("Timed out waiting for service to come alive.")
+
+        # At this point we are guaranteed that the service is alive.
+        self.is_active = True
+
+
+def resolve_classpath(classpath=None):
+    """
+    Returns the classpath to use for corenlp.
+
+    Prefers to use the given classpath parameter, if available.  If
+    not, uses the CORENLP_HOME environment variable.  Resolves $CLASSPATH
+    (the exact string) in either the classpath parameter or $CORENLP_HOME.
+    """
+    if classpath == '$CLASSPATH' or (classpath is None and os.getenv("CORENLP_HOME", None) == '$CLASSPATH'):
+        classpath = os.getenv("CLASSPATH")
+    elif classpath is None:
+        classpath = os.getenv("CORENLP_HOME", os.path.join(str(Path.home()), 'stanza_corenlp'))
+
+        if not os.path.exists(classpath):
+            raise FileNotFoundError("Please install CoreNLP by running `stanza.install_corenlp()`. If you have installed it, please define "
+                                    "$CORENLP_HOME to be location of your CoreNLP distribution or pass in a classpath parameter.  "
+                                    "$CORENLP_HOME={}".format(os.getenv("CORENLP_HOME")))
+        classpath = os.path.join(classpath, "*")
+    return classpath
+
+
+class CoreNLPClient(RobustService):
+    """ A client to the Stanford CoreNLP server. """
+
+    DEFAULT_ENDPOINT = "http://localhost:9000"
+    DEFAULT_TIMEOUT = 60000
+    DEFAULT_THREADS = 5
+    DEFAULT_OUTPUT_FORMAT = "serialized"
+    DEFAULT_MEMORY = "5G"
+    DEFAULT_MAX_CHAR_LENGTH = 100000
+
+    def __init__(self, start_server=StartServer.FORCE_START,
+                 endpoint=DEFAULT_ENDPOINT,
+                 timeout=DEFAULT_TIMEOUT,
+                 threads=DEFAULT_THREADS,
+                 annotators=None,
+                 pretokenized=False,
+                 output_format=None,
+                 properties=None,
+                 stdout=None,
+                 stderr=None,
+                 memory=DEFAULT_MEMORY,
+                 be_quiet=False,
+                 max_char_length=DEFAULT_MAX_CHAR_LENGTH,
+                 preload=True,
+                 classpath=None,
+                 **kwargs):
+
+        # whether or not server should be started by client
+        self.start_server = start_server
+        self.server_props_path = None
+        self.server_start_time = None
+        self.server_host = None
+        self.server_port = None
+        self.server_classpath = None
+        # validate properties
+        validate_corenlp_props(properties=properties, annotators=annotators, output_format=output_format)
+        # set up client defaults
+        self.properties = properties
+        self.annotators = annotators
+        self.pretokenized = pretokenized
+        self.output_format = output_format
+        self._setup_client_defaults()
+        # start the server
+        if isinstance(start_server, bool):
+            warning_msg = f"Setting 'start_server' to a boolean value when constructing {self.__class__.__name__} is deprecated and will stop" + \
+                " to function in a future version of stanza. Please consider switching to using a value from stanza.server.StartServer."
+            logger.warning(warning_msg)
+            start_server = StartServer.FORCE_START if start_server is True else StartServer.DONT_START
+
+        # start the server
+        if start_server is StartServer.FORCE_START or start_server is StartServer.TRY_START:
+            # record info for server start
+            self.server_start_time = datetime.now()
+            # set up default properties for server
+            self._setup_server_defaults()
+            host, port = urlparse(endpoint).netloc.split(":")
+            port = int(port)
+            assert host == "localhost", "If starting a server, endpoint must be localhost"
+            classpath = resolve_classpath(classpath)
+            start_cmd = f"java -Xmx{memory} -cp '{classpath}'  edu.stanford.nlp.pipeline.StanfordCoreNLPServer " \
+                        f"-port {port} -timeout {timeout} -threads {threads} -maxCharLength {max_char_length} " \
+                        f"-quiet {be_quiet} "
+
+            self.server_classpath = classpath
+            self.server_host = host
+            self.server_port = port
+
+            # set up server defaults
+            if self.server_props_path is not None:
+                start_cmd += f" -serverProperties {self.server_props_path}"
+
+            # possibly set pretokenized
+            if self.pretokenized:
+                start_cmd += f" -preTokenized"
+
+            # set annotators for server default
+            if self.annotators is not None:
+                annotators_str = self.annotators if type(annotators) == str else ",".join(annotators)
+                start_cmd += f" -annotators {annotators_str}"
+
+            # specify what to preload, if anything
+            if preload:
+                if type(preload) == bool:
+                    # -preload flag means to preload all default annotators
+                    start_cmd += " -preload"
+                elif type(preload) == list:
+                    # turn list into comma separated list string, only preload these annotators
+                    start_cmd += f" -preload {','.join(preload)}"
+                elif type(preload) == str:
+                    # comma separated list of annotators
+                    start_cmd += f" -preload {preload}"
+
+            # set outputFormat for server default
+            # if no output format requested by user, set to serialized
+            start_cmd += f" -outputFormat {self.output_format}"
+
+            # additional options for server:
+            # - server_id
+            # - ssl
+            # - status_port
+            # - uriContext
+            # - strict
+            # - key
+            # - username
+            # - password
+            # - blockList
+            for kw in ['ssl', 'strict']:
+                if kwargs.get(kw) is not None:
+                    start_cmd += f" -{kw}"
+            for kw in ['status_port', 'uriContext', 'key', 'username', 'password', 'blockList', 'server_id']:
+                if kwargs.get(kw) is not None:
+                    start_cmd += f" -{kw} {kwargs.get(kw)}"
+            stop_cmd = None
+        else:
+            start_cmd = stop_cmd = None
+            host = port = None
+
+        super(CoreNLPClient, self).__init__(start_cmd, stop_cmd, endpoint,
+                                            stdout, stderr, be_quiet, host=host, port=port, ignore_binding_error=(start_server == StartServer.TRY_START))
+
+        self.timeout = timeout
+
+    def _setup_client_defaults(self):
+        """
+        Do some processing of annotators and output_format specified for the client.
+        If interacting with an externally started server, these will be defaults for annotate() calls.
+        :return: None
+        """
+        # normalize annotators to str
+        if self.annotators is not None:
+            self.annotators = self.annotators if type(self.annotators) == str else ",".join(self.annotators)
+
+        # handle case where no output format is specified
+        if self.output_format is None:
+            if type(self.properties) == dict and 'outputFormat' in self.properties:
+                self.output_format = self.properties['outputFormat']
+            else:
+                self.output_format = CoreNLPClient.DEFAULT_OUTPUT_FORMAT
+
+    def _setup_server_defaults(self):
+        """
+        Set up the default properties for the server.
+
+        The properties argument can take on one of 3 value types
+
+        1. File path on system or in CLASSPATH (e.g. /path/to/server.props or StanfordCoreNLP-french.properties
+        2. Name of a Stanford CoreNLP supported language (e.g. french or fr)
+        3. Python dictionary (properties written to tmp file for Java server, erased at end)
+
+        In addition, an annotators list and output_format can be specified directly with arguments. These
+        will overwrite any settings in the specified properties.
+
+        If no properties are specified, the standard Stanford CoreNLP English server will be launched. The outputFormat
+        will be set to 'serialized' and use the ProtobufAnnotationSerializer.
+        """
+
+        # ensure properties is str or dict
+        if self.properties is None or (not isinstance(self.properties, str) and not isinstance(self.properties, dict)):
+            if self.properties is not None:
+                logger.warning('properties passed invalid value (not a str or dict), setting properties = {}')
+            self.properties = {}
+        # check if properties is a string, pass on to server which can handle
+        if isinstance(self.properties, str):
+            # try to translate to Stanford CoreNLP language name, or assume properties is a path
+            if is_corenlp_lang(self.properties):
+                if self.properties.lower() in LANGUAGE_SHORTHANDS_TO_FULL:
+                    self.properties = LANGUAGE_SHORTHANDS_TO_FULL[self.properties]
+                logger.info(
+                    f"Using CoreNLP default properties for: {self.properties}.  Make sure to have "
+                    f"{self.properties} models jar (available for download here: "
+                    f"https://stanfordnlp.github.io/CoreNLP/) in CLASSPATH")
+            else:
+                if not os.path.isfile(self.properties):
+                    logger.warning(f"{self.properties} does not correspond to a file path. Make sure this file is in "
+                                   f"your CLASSPATH.")
+            self.server_props_path = self.properties
+        elif isinstance(self.properties, dict):
+            # make a copy
+            server_start_properties = dict(self.properties)
+            if self.annotators is not None:
+                server_start_properties['annotators'] = self.annotators
+            if self.output_format is not None and isinstance(self.output_format, str):
+                server_start_properties['outputFormat'] = self.output_format
+            # write desired server start properties to tmp file
+            # set up to erase on exit
+            tmp_path = write_corenlp_props(server_start_properties)
+            logger.info(f"Writing properties to tmp file: {tmp_path}")
+            atexit.register(clean_props_file, tmp_path)
+            self.server_props_path = tmp_path
+
+    def _request(self, buf, properties, reset_default=False, **kwargs):
+        """
+        Send a request to the CoreNLP server.
+
+        :param (str | bytes) buf: data to be sent with the request
+        :param (dict) properties: properties that the server expects
+        :return: request result
+        """
+        if self.start_server is not StartServer.DONT_START:
+            self.ensure_alive()
+
+        try:
+            input_format = properties.get("inputFormat", "text")
+            if input_format == "text":
+                ctype = "text/plain; charset=utf-8"
+            elif input_format == "serialized":
+                ctype = "application/x-protobuf"
+            else:
+                raise ValueError("Unrecognized inputFormat " + input_format)
+            # handle auth
+            if 'username' in kwargs and 'password' in kwargs:
+                kwargs['auth'] = requests.auth.HTTPBasicAuth(kwargs['username'], kwargs['password'])
+                kwargs.pop('username')
+                kwargs.pop('password')
+            r = requests.post(self.endpoint,
+                              params={'properties': str(properties), 'resetDefault': str(reset_default).lower()},
+                              data=buf, headers={'content-type': ctype},
+                              timeout=(self.timeout*2)/1000, **kwargs)
+            r.raise_for_status()
+            return r
+        except requests.exceptions.Timeout as e:
+            raise TimeoutException("Timeout requesting to CoreNLPServer. Maybe server is unavailable or your document is too long")
+        except requests.exceptions.RequestException as e:
+            if e.response is not None and e.response.text is not None:
+                raise AnnotationException(e.response.text) from e
+            elif e.args:
+                raise AnnotationException(e.args[0]) from e
+            raise AnnotationException() from e
+
+    def annotate(self, text, annotators=None, output_format=None, properties=None, reset_default=None, **kwargs):
+        """
+        Send a request to the CoreNLP server.
+
+        :param (str | unicode) text: raw text for the CoreNLPServer to parse
+        :param (list | string) annotators: list of annotators to use
+        :param (str) output_format: output type from server: serialized, json, text, conll, conllu, or xml
+        :param (dict) properties: additional request properties (written on top of defaults)
+        :param (bool) reset_default: don't use server defaults
+
+        Precedence for settings:
+
+        1. annotators and output_format args
+        2. Values from properties dict
+        3. Client defaults self.annotators and self.output_format (set during client construction)
+        4. Server defaults
+
+        Additional request parameters (apart from CoreNLP pipeline properties) such as 'username' and 'password'
+        can be specified with the kwargs.
+
+        :return: request result
+        """
+
+        # validate request properties
+        validate_corenlp_props(properties=properties, annotators=annotators, output_format=output_format)
+        # set request properties
+        request_properties = {}
+
+        # start with client defaults
+        if self.annotators is not None:
+            request_properties['annotators'] = self.annotators
+        if self.output_format is not None:
+            request_properties['outputFormat'] = self.output_format
+
+        # add values from properties arg
+        # handle str case
+        if type(properties) == str:
+            if is_corenlp_lang(properties):
+                properties = {'pipelineLanguage': properties.lower()}
+                if reset_default is None:
+                    reset_default = True
+            else:
+                raise ValueError(f"Unrecognized properties keyword {properties}")
+
+        if type(properties) == dict:
+            request_properties.update(properties)
+
+        # if annotators list is specified, override with that
+        # also can use the annotators field the object was created with
+        if annotators is not None and (type(annotators) == str or type(annotators) == list):
+            request_properties['annotators'] = annotators if type(annotators) == str else ",".join(annotators)
+
+        # if output format is specified, override with that
+        if output_format is not None and type(output_format) == str:
+            request_properties['outputFormat'] = output_format
+
+        # make the request
+        # if not explicitly set or the case of pipelineLanguage, reset_default should be None
+        if reset_default is None:
+            reset_default = False
+        r = self._request(text.encode('utf-8'), request_properties, reset_default, **kwargs)
+        if request_properties["outputFormat"] == "json":
+            return r.json()
+        elif request_properties["outputFormat"] == "serialized":
+            doc = Document()
+            parseFromDelimitedString(doc, r.content)
+            return doc
+        elif request_properties["outputFormat"] in ["text", "conllu", "conll", "xml"]:
+            return r.text
+        else:
+            return r
+
+    def update(self, doc, annotators=None, properties=None):
+        if properties is None:
+            properties = {}
+            properties.update({
+                'inputFormat': 'serialized',
+                'outputFormat': 'serialized',
+                'serializer': 'edu.stanford.nlp.pipeline.ProtobufAnnotationSerializer'
+            })
+        if annotators:
+            properties['annotators'] = annotators if type(annotators) == str else ",".join(annotators)
+        with io.BytesIO() as stream:
+            writeToDelimitedString(doc, stream)
+            msg = stream.getvalue()
+
+        r = self._request(msg, properties)
+        doc = Document()
+        parseFromDelimitedString(doc, r.content)
+        return doc
+
+    def tokensregex(self, text, pattern, filter=False, to_words=False, annotators=None, properties=None):
+        # this is required for some reason
+        matches = self.__regex('/tokensregex', text, pattern, filter, annotators, properties)
+        if to_words:
+            matches = regex_matches_to_indexed_words(matches)
+        return matches
+
+    def semgrex(self, text, pattern, filter=False, to_words=False, annotators=None, properties=None):
+        matches = self.__regex('/semgrex', text, pattern, filter, annotators, properties)
+        if to_words:
+            matches = regex_matches_to_indexed_words(matches)
+        return matches
+
+    def fill_tree_proto(self, tree, proto_tree):
+        if tree.label:
+            proto_tree.value = tree.label
+        for child in tree.children:
+            proto_child = proto_tree.child.add()
+            self.fill_tree_proto(child, proto_child)
+
+    def tregex(self, text=None, pattern=None, filter=False, annotators=None, properties=None, trees=None):
+        # parse is not included by default in some of the pipelines,
+        # so we may need to manually override the annotators
+        # to include parse in order for tregex to do anything
+        if annotators is None and self.annotators is not None:
+            assert isinstance(self.annotators, str)
+            pieces = self.annotators.split(",")
+            if "parse" not in pieces:
+                annotators = self.annotators + ",parse"
+        else:
+            annotators = "tokenize,ssplit,pos,parse"
+        if pattern is None:
+            raise ValueError("Cannot have None as a pattern for tregex")
+
+        # TODO: we could also allow for passing in a complete document,
+        # along with the original text, so that the spans returns are more accurate
+        if trees is not None:
+            if properties is None:
+                properties = {}
+            properties['inputFormat'] = 'serialized'
+            if 'serializer' not in properties:
+                properties['serializer'] = 'edu.stanford.nlp.pipeline.ProtobufAnnotationSerializer'
+            doc = Document()
+            full_text = []
+            for tree_idx, tree in enumerate(trees):
+                sentence = doc.sentence.add()
+                sentence.sentenceIndex = tree_idx
+                sentence.tokenOffsetBegin = len(full_text)
+                leaves = tree.leaf_labels()
+                full_text.extend(leaves)
+                sentence.tokenOffsetEnd = len(full_text)
+                self.fill_tree_proto(tree, sentence.parseTree)
+                for word in leaves:
+                    token = sentence.token.add()
+                    # the other side uses both value and word, weirdly enough
+                    token.value = word
+                    token.word = word
+                    # without the actual tokenization, at least we can
+                    # stop the words from running together
+                    token.after = " "
+            doc.text = " ".join(full_text)
+            with io.BytesIO() as stream:
+                writeToDelimitedString(doc, stream)
+                text = stream.getvalue()
+
+        return self.__regex('/tregex', text, pattern, filter, annotators, properties)
+
+    def __regex(self, path, text, pattern, filter, annotators=None, properties=None):
+        """
+        Send a regex-related request to the CoreNLP server.
+
+        :param (str | unicode) path: the path for the regex endpoint
+        :param text: raw text for the CoreNLPServer to apply the regex
+        :param (str | unicode) pattern: regex pattern
+        :param (bool) filter: option to filter sentences that contain matches, if false returns matches
+        :param properties: option to filter sentences that contain matches, if false returns matches
+        :return: request result
+        """
+        if self.start_server is not StartServer.DONT_START:
+            self.ensure_alive()
+        if properties is None:
+            properties = {}
+            properties.update({
+                'inputFormat': 'text',
+                'serializer': 'edu.stanford.nlp.pipeline.ProtobufAnnotationSerializer'
+            })
+        if annotators:
+            properties['annotators'] = ",".join(annotators) if isinstance(annotators, list) else annotators
+
+        # force output for regex requests to be json
+        properties['outputFormat'] = 'json'
+        # if the server is trying to send back character offsets, it
+        # should send back codepoints counts as well in case the text
+        # has extra wide characters
+        properties['tokenize.codepoint'] = 'true'
+
+        try:
+            # Error occurs unless put properties in params
+            input_format = properties.get("inputFormat", "text")
+            if input_format == "text":
+                ctype = "text/plain; charset=utf-8"
+            elif input_format == "serialized":
+                ctype = "application/x-protobuf"
+            else:
+                raise ValueError("Unrecognized inputFormat " + input_format)
+            # change request method from `get` to `post` as required by CoreNLP
+            r = requests.post(
+                self.endpoint + path, params={
+                    'pattern': pattern,
+                    'filter': filter,
+                    'properties': str(properties)
+                },
+                data=text.encode('utf-8') if isinstance(text, str) else text,
+                headers={'content-type': ctype},
+                timeout=(self.timeout*2)/1000,
+            )
+            r.raise_for_status()
+            if r.encoding is None:
+                r.encoding = "utf-8"
+            return json.loads(r.text)
+        except requests.HTTPError as e:
+            if r.text.startswith("Timeout"):
+                raise TimeoutException(r.text)
+            else:
+                raise AnnotationException(r.text)
+        except json.JSONDecodeError:
+            raise AnnotationException(r.text)
+
+
+    def scenegraph(self, text, properties=None):
+        """
+        Send a request to the server which processes the text using SceneGraph
+
+        This will require a new CoreNLP release, 4.5.5 or later
+        """
+        # since we're using requests ourself,
+        # check if the server has started or not
+        if self.start_server is not StartServer.DONT_START:
+            self.ensure_alive()
+
+        if properties is None:
+            properties = {}
+        # the only thing the scenegraph knows how to use is text
+        properties['inputFormat'] = 'text'
+        ctype = "text/plain; charset=utf-8"
+        # the json output format is much more useful
+        properties['outputFormat'] = 'json'
+        try:
+            r = requests.post(
+                self.endpoint + "/scenegraph",
+                params={
+                    'properties': str(properties)
+                },
+                data=text.encode('utf-8') if isinstance(text, str) else text,
+                headers={'content-type': ctype},
+                timeout=(self.timeout*2)/1000,
+            )
+            r.raise_for_status()
+            if r.encoding is None:
+                r.encoding = "utf-8"
+            return json.loads(r.text)
+        except requests.HTTPError as e:
+            if r.text.startswith("Timeout"):
+                raise TimeoutException(r.text)
+            else:
+                raise AnnotationException(r.text)
+        except json.JSONDecodeError:
+            raise AnnotationException(r.text)
+
+
+def read_corenlp_props(props_path):
+    """ Read a Stanford CoreNLP properties file into a dict """
+    props_dict = {}
+    with open(props_path) as props_file:
+        entry_lines = [entry_line for entry_line in props_file.read().split('\n')
+                       if entry_line.strip() and not entry_line.startswith('#')]
+        for entry_line in entry_lines:
+            k = entry_line.split('=')[0]
+            k_len = len(k+"=")
+            v = entry_line[k_len:]
+            props_dict[k.strip()] = v
+    return props_dict
+
+
+def write_corenlp_props(props_dict, file_path=None):
+    """ Write a Stanford CoreNLP properties dict to a file """
+    if file_path is None:
+        file_path = f"corenlp_server-{uuid.uuid4().hex[:16]}.props"
+        # confirm tmp file path matches pattern
+        assert SERVER_PROPS_TMP_FILE_PATTERN.match(file_path)
+    with open(file_path, 'w') as props_file:
+        for k, v in props_dict.items():
+            if isinstance(v, list):
+                writeable_v = ",".join(v)
+            else:
+                writeable_v = v
+            props_file.write(f'{k} = {writeable_v}\n\n')
+    return file_path
+
+
+def regex_matches_to_indexed_words(matches):
+    """
+    Transforms tokensregex and semgrex matches to indexed words.
+    :param matches: unprocessed regex matches
+    :return: flat array of indexed words
+    """
+    words = [dict(v, **dict([('sentence', i)]))
+             for i, s in enumerate(matches['sentences'])
+             for k, v in s.items() if k != 'length']
+    return words
+
+
+__all__ = ["CoreNLPClient", "AnnotationException", "TimeoutException", "to_text"]

stanza/stanza/server/semgrex.py

@@ -0,0 +1,170 @@
+"""Invokes the Java semgrex on a document
+
+The server client has a method "semgrex" which sends text to Java
+CoreNLP for processing with a semgrex (SEMantic GRaph regEX) query:
+
+https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/semgraph/semgrex/SemgrexPattern.html
+
+However, this operates on text using the CoreNLP tools, which means
+the dependency graphs may not align with stanza's depparse module, and
+this also limits the languages for which it can be used.  This module
+allows for running semgrex commands on the graphs produced by
+depparse.
+
+To use, first process text into a doc using stanza.Pipeline
+
+Next, pass the processed doc and a list of semgrex patterns to
+process_doc in this module.  It will run the java semgrex module as a
+subprocess and return the result in the form of a SemgrexResponse,
+whose description is in the proto file included with stanza.
+
+A minimal example is the main method of this module.
+
+Note that launching the subprocess is potentially quite expensive
+relative to the search if used many times on small documents.  Ideally
+larger texts would be processed, and all of the desired semgrex
+patterns would be run at once.  The worst thing to do would be to call
+this multiple times on a large document, one invocation per semgrex
+pattern, as that would serialize the document each time.
+Included here is a context manager which allows for keeping the same
+java process open for multiple requests.  This saves on the subprocess
+launching time.  It is still important not to wastefully serialize the
+same document over and over, though.
+"""
+
+import argparse
+import copy
+
+import stanza
+from stanza.protobuf import SemgrexRequest, SemgrexResponse
+from stanza.server.java_protobuf_requests import send_request, add_token, add_word_to_graph, JavaProtobufContext, convert_networkx_graph
+from stanza.utils.conll import CoNLL
+
+SEMGREX_JAVA = "edu.stanford.nlp.semgraph.semgrex.ProcessSemgrexRequest"
+
+def send_semgrex_request(request):
+    return send_request(request, SemgrexResponse, SEMGREX_JAVA)
+
+def build_request(doc, semgrex_patterns, enhanced=False):
+    request = SemgrexRequest()
+    if isinstance(semgrex_patterns, str):
+        semgrex_patterns = [semgrex_patterns]
+    for semgrex in semgrex_patterns:
+        request.semgrex.append(semgrex)
+
+    for sent_idx, sentence in enumerate(doc.sentences):
+        query = request.query.add()
+        if enhanced:
+            # tokens will be added on to the graph object
+            convert_networkx_graph(query.graph, sentence, sent_idx)
+        else:
+            word_idx = 0
+            for token in sentence.tokens:
+                for word in token.words:
+                    add_token(query.token, word, token)
+                    add_word_to_graph(query.graph, word, sent_idx, word_idx)
+
+                    word_idx = word_idx + 1
+
+    return request
+
+def process_doc(doc, *semgrex_patterns, enhanced=False):
+    """
+    Returns the result of processing the given semgrex expression on the stanza doc.
+
+    Currently the return is a SemgrexResponse from CoreNLP.proto
+    """
+    request = build_request(doc, semgrex_patterns, enhanced=enhanced)
+
+    return send_semgrex_request(request)
+
+class Semgrex(JavaProtobufContext):
+    """
+    Semgrex context window
+
+    This is a context window which keeps a process open.  Should allow
+    for multiple requests without launching new java processes each time.
+    """
+    def __init__(self, classpath=None):
+        super(Semgrex, self).__init__(classpath, SemgrexResponse, SEMGREX_JAVA)
+
+    def process(self, doc, *semgrex_patterns):
+        """
+        Apply each of the semgrex patterns to each of the dependency trees in doc
+        """
+        request = build_request(doc, semgrex_patterns)
+        return self.process_request(request)
+
+def annotate_doc(doc, semgrex_result, semgrex_patterns, matches_only):
+    """
+    Put comments on the sentences which describe the matching semgrex patterns
+    """
+    doc = copy.deepcopy(doc)
+    if isinstance(semgrex_patterns, str):
+        semgrex_patterns = [semgrex_patterns]
+    matching_sentences = []
+    for sentence, graph_result in zip(doc.sentences, semgrex_result.result):
+        sentence_matched = False
+        for semgrex_pattern, pattern_result in zip(semgrex_patterns, graph_result.result):
+            semgrex_pattern = semgrex_pattern.replace("\n", " ")
+            if len(pattern_result.match) == 0:
+                sentence.add_comment("# semgrex pattern |%s| did not match!" % semgrex_pattern)
+            else:
+                sentence_matched = True
+                for match in pattern_result.match:
+                    match_word = "%d:%s" % (match.matchIndex, sentence.words[match.matchIndex-1].text)
+                    if len(match.node) == 0:
+                        node_matches = ""
+                    else:
+                        node_matches = ["%s=%d:%s" % (node.name, node.matchIndex, sentence.words[node.matchIndex-1].text)
+                                        for node in match.node]
+                        node_matches = "  " + " ".join(node_matches)
+                    sentence.add_comment("# semgrex pattern |%s| matched at %s%s" % (semgrex_pattern, match_word, node_matches))
+        if sentence_matched:
+            matching_sentences.append(sentence)
+    if matches_only:
+        doc.sentences = matching_sentences
+    return doc
+
+
+def main():
+    """
+    Runs a toy example, or can run a given semgrex expression on the given input file.
+
+    For example:
+    python3 -m stanza.server.semgrex --input_file demo/semgrex_sample.conllu
+
+    --matches_only to only print sentences that match the semgrex pattern
+    --no_print_input to not print the input
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--input_file', type=str, default=None, help="Input file to process (otherwise will process a sample text)")
+    parser.add_argument('semgrex', type=str, nargs="*", default=["{}=source >obj=zzz {}=target"], help="Semgrex to apply to the text.  The default looks for sentences with objects")
+    parser.add_argument('--semgrex_file', type=str, default=None, help="File to read semgrex patterns from - relevant in case the pattern you want to use doesn't work well on the command line, for example")
+    parser.add_argument('--print_input', dest='print_input', action='store_true', default=False, help="Print the input alongside the output - gets kind of noisy")
+    parser.add_argument('--no_print_input', dest='print_input', action='store_false', help="Don't print the input alongside the output - gets kind of noisy")
+    parser.add_argument('--matches_only', action='store_true', default=False, help="Only print the matching sentences")
+    parser.add_argument('--enhanced', action='store_true', default=False, help='Use the enhanced dependencies instead of the basic')
+    args = parser.parse_args()
+
+    if args.semgrex_file:
+        with open(args.semgrex_file) as fin:
+            args.semgrex = [x.strip() for x in fin.readlines() if x.strip()]
+
+    if args.input_file:
+        doc = CoNLL.conll2doc(input_file=args.input_file, ignore_gapping=False)
+    else:
+        nlp = stanza.Pipeline('en', processors='tokenize,pos,lemma,depparse')
+        doc = nlp('Uro ruined modern.  Fortunately, Wotc banned him.')
+
+    if args.print_input:
+        print("{:C}".format(doc))
+        print()
+        print("-" * 75)
+        print()
+    semgrex_result = process_doc(doc, *args.semgrex, enhanced=args.enhanced)
+    doc = annotate_doc(doc, semgrex_result, args.semgrex, args.matches_only)
+    print("{:C}".format(doc))
+
+if __name__ == '__main__':
+    main()

stanza/stanza/server/ssurgeon.py

@@ -0,0 +1,310 @@
+"""Invokes the Java ssurgeon on a document
+
+"ssurgeon" sends text to Java CoreNLP for processing with a ssurgeon
+(Semantic graph SURGEON) query
+
+The main program in this file gives a very short intro to how to use it.
+"""
+
+
+import argparse
+from collections import namedtuple
+import copy
+import os
+import re
+import sys
+
+from stanza.models.common.utils import misc_to_space_after, space_after_to_misc
+from stanza.protobuf import SsurgeonRequest, SsurgeonResponse
+from stanza.server import java_protobuf_requests
+from stanza.utils.conll import CoNLL
+
+from stanza.models.common.doc import ID, TEXT, LEMMA, UPOS, XPOS, FEATS, HEAD, DEPREL, DEPS, MISC, START_CHAR, END_CHAR, NER, Word, Token, Sentence
+
+SSURGEON_JAVA = "edu.stanford.nlp.semgraph.semgrex.ssurgeon.ProcessSsurgeonRequest"
+
+SsurgeonEdit = namedtuple("SsurgeonEdit",
+                          "semgrex_pattern ssurgeon_edits ssurgeon_id notes language",
+                          defaults=[None, None, "UniversalEnglish"])
+
+def parse_ssurgeon_edits(ssurgeon_text):
+    ssurgeon_text = ssurgeon_text.strip()
+    ssurgeon_blocks = re.split("\n\n+", ssurgeon_text)
+    ssurgeon_edits = []
+    for idx, block in enumerate(ssurgeon_blocks):
+        lines = block.split("\n")
+        comments = [line[1:].strip() for line in lines if line.startswith("#")]
+        notes = " ".join(comments)
+        lines = [x.strip() for x in lines if x.strip() and not x.startswith("#")]
+        if len(lines) == 0:
+            # was a block of entirely comments
+            continue
+        semgrex = lines[0]
+        ssurgeon = lines[1:]
+        ssurgeon_edits.append(SsurgeonEdit(semgrex, ssurgeon, "%d" % (idx + 1), notes))
+    return ssurgeon_edits
+
+def read_ssurgeon_edits(edit_file):
+    with open(edit_file, encoding="utf-8") as fin:
+        return parse_ssurgeon_edits(fin.read())
+
+def send_ssurgeon_request(request):
+    return java_protobuf_requests.send_request(request, SsurgeonResponse, SSURGEON_JAVA)
+
+def build_request(doc, ssurgeon_edits):
+    request = SsurgeonRequest()
+
+    for ssurgeon in ssurgeon_edits:
+        ssurgeon_proto = request.ssurgeon.add()
+        ssurgeon_proto.semgrex = ssurgeon.semgrex_pattern
+        for operation in ssurgeon.ssurgeon_edits:
+            ssurgeon_proto.operation.append(operation)
+        if ssurgeon.ssurgeon_id is not None:
+            ssurgeon_proto.id = ssurgeon.ssurgeon_id
+        if ssurgeon.notes is not None:
+            ssurgeon_proto.notes = ssurgeon.notes
+        if ssurgeon.language is not None:
+            ssurgeon_proto.language = ssurgeon.language
+
+    try:
+        for sent_idx, sentence in enumerate(doc.sentences):
+            graph = request.graph.add()
+            word_idx = 0
+            for token in sentence.tokens:
+                for word in token.words:
+                    java_protobuf_requests.add_token(graph.token, word, token)
+                    java_protobuf_requests.add_word_to_graph(graph, word, sent_idx, word_idx)
+
+                    word_idx = word_idx + 1
+    except Exception as e:
+        raise RuntimeError("Failed to process sentence {}:\n{:C}".format(sent_idx, sentence)) from e
+
+    return request
+
+def build_request_one_operation(doc, semgrex_pattern, ssurgeon_edits, ssurgeon_id=None, notes=None):
+    ssurgeon_edit = SsurgeonEdit(semgrex_pattern, ssurgeon_edits, ssurgeon_id, notes)
+    return build_request(doc, [ssurgeon_edit])
+
+def process_doc(doc, ssurgeon_edits):
+    """
+    Returns the result of processing the given semgrex expression and ssurgeon edits on the stanza doc.
+
+    Currently the return is a SsurgeonResponse from CoreNLP.proto
+    """
+    request = build_request(doc, ssurgeon_edits)
+
+    return send_ssurgeon_request(request)
+
+def process_doc_one_operation(doc, semgrex_pattern, ssurgeon_edits, ssurgeon_id=None, notes=None):
+    request = build_request_one_operation(doc, semgrex_pattern, ssurgeon_edits, ssurgeon_id, notes)
+
+    return send_ssurgeon_request(request)
+
+def build_word_entry(word_index, graph_word):
+    word_entry = {
+        ID: word_index,
+        TEXT: graph_word.word if graph_word.word else None,
+        LEMMA: graph_word.lemma if graph_word.lemma else None,
+        UPOS: graph_word.coarseTag if graph_word.coarseTag else None,
+        XPOS: graph_word.pos if graph_word.pos else None,
+        FEATS: java_protobuf_requests.features_to_string(graph_word.conllUFeatures),
+        DEPS: None,
+        NER: graph_word.ner if graph_word.ner else None,
+        MISC: None,
+        START_CHAR: None,   # TODO: fix this?  one problem is the text positions
+        END_CHAR: None,     #   might change across all of the sentences
+        # presumably python will complain if this conflicts
+        # with one of the constants above
+        "is_mwt": graph_word.isMWT,
+        "is_first_mwt": graph_word.isFirstMWT,
+        "mwt_text": graph_word.mwtText,
+        "mwt_misc": graph_word.mwtMisc,
+    }
+    # TODO: do "before" as well
+    word_entry[MISC] = space_after_to_misc(graph_word.after)
+    if graph_word.conllUMisc:
+        word_entry[MISC] = java_protobuf_requests.substitute_space_misc(graph_word.conllUMisc, word_entry[MISC])
+    return word_entry
+
+def convert_response_to_doc(doc, semgrex_response):
+    doc = copy.deepcopy(doc)
+    try:
+        for sent_idx, (sentence, ssurgeon_result) in enumerate(zip(doc.sentences, semgrex_response.result)):
+            # EditNode is currently bugged... :/
+            # TODO: change this after next CoreNLP release (after 4.5.3)
+            #if not ssurgeon_result.changed:
+            #    continue
+
+            ssurgeon_graph = ssurgeon_result.graph
+            tokens = []
+            for graph_node, graph_word in zip(ssurgeon_graph.node, ssurgeon_graph.token):
+                word_entry = build_word_entry(graph_node.index, graph_word)
+                tokens.append(word_entry)
+            tokens.sort(key=lambda x: x[ID])
+            for root in ssurgeon_graph.root:
+                tokens[root-1][HEAD] = 0
+                tokens[root-1][DEPREL] = "root"
+            for edge in ssurgeon_graph.edge:
+                # can't do anything about the extra dependencies for now
+                # TODO: put them all in .deps
+                if edge.isExtra:
+                    continue
+                tokens[edge.target-1][HEAD] = edge.source
+                tokens[edge.target-1][DEPREL] = edge.dep
+
+            # for any MWT, produce a token_entry which represents the word range
+            mwt_tokens = []
+            for word_start_idx, word in enumerate(tokens):
+                if not word["is_first_mwt"]:
+                    if word["is_mwt"]:
+                        word[MISC] = java_protobuf_requests.remove_space_misc(word[MISC])
+                    mwt_tokens.append(word)
+                    continue
+                word_end_idx = word_start_idx + 1
+                while word_end_idx < len(tokens) and tokens[word_end_idx]["is_mwt"] and not tokens[word_end_idx]["is_first_mwt"]:
+                    word_end_idx += 1
+                mwt_token_entry = {
+                    # the tokens don't fencepost the way lists do
+                    ID: (tokens[word_start_idx][ID], tokens[word_end_idx-1][ID]),
+                    TEXT: word["mwt_text"],
+                    NER: word[NER],
+                    # use the SpaceAfter=No (or not) from the last word in the token
+                    MISC: None,
+                }
+                mwt_token_entry[MISC] = java_protobuf_requests.misc_space_pieces(tokens[word_end_idx-1][MISC])
+                if tokens[word_end_idx-1]["mwt_misc"]:
+                    mwt_token_entry[MISC] = java_protobuf_requests.substitute_space_misc(tokens[word_end_idx-1]["mwt_misc"], mwt_token_entry[MISC])
+                word[MISC] = java_protobuf_requests.remove_space_misc(word[MISC])
+                mwt_tokens.append(mwt_token_entry)
+                mwt_tokens.append(word)
+
+            old_comments = list(sentence.comments)
+            sentence = Sentence(mwt_tokens, doc)
+
+            token_text = []
+            for token_idx, token in enumerate(sentence.tokens):
+                token_text.append(token.text)
+                if token_idx == len(sentence.tokens) - 1:
+                    break
+                token_text.append(token.spaces_after)
+
+            sentence_text = "".join(token_text)
+
+            for comment in old_comments:
+                if comment.startswith("# text ") or comment.startswith("#text ") or comment.startswith("# text=") or comment.startswith("#text="):
+                    sentence.add_comment("# text = " + sentence_text)
+                else:
+                    sentence.add_comment(comment)
+
+            doc.sentences[sent_idx] = sentence
+
+            sentence.rebuild_dependencies()
+    except Exception as e:
+        raise RuntimeError("Ssurgeon could not process sentence {}\nSsurgeon result:\n{}\nOriginal sentence:\n{:C}".format(sent_idx, ssurgeon_result, sentence)) from e
+    return doc
+
+class Ssurgeon(java_protobuf_requests.JavaProtobufContext):
+    """
+    Ssurgeon context window
+
+    This is a context window which keeps a process open.  Should allow
+    for multiple requests without launching new java processes each time.
+    """
+    def __init__(self, classpath=None):
+        super(Ssurgeon, self).__init__(classpath, SsurgeonResponse, SSURGEON_JAVA)
+
+    def process(self, doc, ssurgeon_edits):
+        """
+        Apply each of the ssurgeon patterns to each of the dependency trees in doc
+        """
+        request = build_request(doc, ssurgeon_edits)
+        return self.process_request(request)
+
+    def process_one_operation(self, doc, semgrex_pattern, ssurgeon_edits, ssurgeon_id=None, notes=None):
+        """
+        Convenience method - build one operation, then apply it
+        """
+        request = build_request_one_operation(doc, semgrex_pattern, ssurgeon_edits, ssurgeon_id, notes)
+        return self.process_request(request)
+
+SAMPLE_DOC = """
+# sent_id = 271
+# text = Hers is easy to clean.
+# previous = What did the dealer like about Alex's car?
+# comment = extraction/raising via "tough extraction" and clausal subject
+1	Hers	hers	PRON	PRP	Gender=Fem|Number=Sing|Person=3|Poss=Yes|PronType=Prs	3	nsubj	_	_
+2	is	be	AUX	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	3	cop	_	_
+3	easy	easy	ADJ	JJ	Degree=Pos	0	root	_	_
+4	to	to	PART	TO	_	5	mark	_	_
+5	clean	clean	VERB	VB	VerbForm=Inf	3	csubj	_	SpaceAfter=No
+6	.	.	PUNCT	.	_	5	punct	_	_
+"""
+
+def main():
+    # for Windows, so that we aren't randomly printing garbage (or just failing to print)
+    try:
+        sys.stdout.reconfigure(encoding='utf-8')
+    except AttributeError:
+        # TODO: deprecate 3.6 support after the next release
+        pass
+
+    # The default semgrex detects sentences in the UD_English-Pronouns dataset which have both nsubj and csubj on the same word.
+    # The default ssurgeon transforms the unwanted csubj to advcl
+    # See https://github.com/UniversalDependencies/docs/issues/923
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--input_file', type=str, default=None, help="Input file to process (otherwise will process a sample text)")
+    parser.add_argument('--output_file', type=str, default=None, help="Output file (otherwise will write to stdout)")
+    parser.add_argument('--input_dir', type=str, default=None, help="Input dir to process instead of a single file.  Allows for reusing the Java program")
+    parser.add_argument('--input_filter', type=str, default=".*[.]conllu", help="Only process files from the input_dir that match this filter - regex, not shell filter.  Default: %(default)s")
+    parser.add_argument('--no_input_filter', action='store_const', const=None, help="Remove the default input filename filter")
+    parser.add_argument('--output_dir', type=str, default=None, help="Output dir for writing files, necessary if using --input_dir")
+    parser.add_argument('--edit_file', type=str, default=None, help="File to get semgrex and ssurgeon rules from")
+    parser.add_argument('--semgrex', type=str, default="{}=source >nsubj {} >csubj=bad {}", help="Semgrex to apply to the text.  A default detects words which have both an nsubj and a csubj.  Default: %(default)s")
+    parser.add_argument('ssurgeon', type=str, default=["relabelNamedEdge -edge bad -reln advcl"], nargs="*", help="Ssurgeon edits to apply based on the Semgrex.  Can have multiple edits in a row.  A default exists to transform csubj into advcl.  Default: %(default)s")
+    parser.add_argument('--print_input', dest='print_input', action='store_true', default=False, help="Print the input alongside the output - gets kind of noisy.  Default: %(default)s")
+    parser.add_argument('--no_print_input', dest='print_input', action='store_false', help="Don't print the input alongside the output - gets kind of noisy")
+    args = parser.parse_args()
+
+    if args.edit_file:
+        ssurgeon_edits = read_ssurgeon_edits(args.edit_file)
+    else:
+        ssurgeon_edits = [SsurgeonEdit(args.semgrex, args.ssurgeon)]
+
+    if args.input_file:
+        docs = [CoNLL.conll2doc(input_file=args.input_file)]
+        outputs = [args.output_file]
+        input_output = zip(docs, outputs)
+    elif args.input_dir:
+        if not args.output_dir:
+            raise ValueError("Cannot process multiple files without knowing where to send them - please set --output_dir in order to use --input_dir")
+        if not os.path.exists(args.output_dir):
+            os.makedirs(args.output_dir)
+        def read_docs():
+            for doc_filename in os.listdir(args.input_dir):
+                if args.input_filter:
+                    if not re.match(args.input_filter, doc_filename):
+                        continue
+                doc_path = os.path.join(args.input_dir, doc_filename)
+                output_path = os.path.join(args.output_dir, doc_filename)
+                print("Processing %s to %s" % (doc_path, output_path))
+                yield CoNLL.conll2doc(input_file=doc_path), output_path
+        input_output = read_docs()
+    else:
+        docs = [CoNLL.conll2doc(input_str=SAMPLE_DOC)]
+        outputs = [None]
+        input_output = zip(docs, outputs)
+
+    for doc, output in input_output:
+        if args.print_input:
+            print("{:C}".format(doc))
+        ssurgeon_request = build_request(doc, ssurgeon_edits)
+        ssurgeon_response = send_ssurgeon_request(ssurgeon_request)
+        updated_doc = convert_response_to_doc(doc, ssurgeon_response)
+        if output is not None:
+            with open(output, "w", encoding="utf-8") as fout:
+                fout.write("{:C}\n\n".format(updated_doc))
+        else:
+            print("{:C}\n".format(updated_doc))
+
+if __name__ == '__main__':
+    main()

stanza/stanza/tests/__init__.py

@@ -0,0 +1,111 @@
+"""
+Utilities for testing
+"""
+
+import os
+import re
+
+# Environment Variables
+# set this to specify working directory of tests
+TEST_HOME_VAR = 'STANZA_TEST_HOME'
+
+# Global Variables
+TEST_DIR_BASE_NAME = 'stanza_test'
+
+TEST_WORKING_DIR = os.getenv(TEST_HOME_VAR, None)
+if not TEST_WORKING_DIR:
+    TEST_WORKING_DIR = os.path.join(os.getcwd(), TEST_DIR_BASE_NAME)
+
+TEST_MODELS_DIR = f'{TEST_WORKING_DIR}/models'
+TEST_CORENLP_DIR = f'{TEST_WORKING_DIR}/corenlp_dir'
+
+# server resources
+SERVER_TEST_PROPS = f'{TEST_WORKING_DIR}/scripts/external_server.properties'
+
+# language resources
+LANGUAGE_RESOURCES = {}
+
+TOKENIZE_MODEL = 'tokenizer.pt'
+MWT_MODEL = 'mwt_expander.pt'
+POS_MODEL = 'tagger.pt'
+POS_PRETRAIN = 'pretrain.pt'
+LEMMA_MODEL = 'lemmatizer.pt'
+DEPPARSE_MODEL = 'parser.pt'
+DEPPARSE_PRETRAIN = 'pretrain.pt'
+
+MODEL_FILES = [TOKENIZE_MODEL, MWT_MODEL, POS_MODEL, POS_PRETRAIN, LEMMA_MODEL, DEPPARSE_MODEL, DEPPARSE_PRETRAIN]
+
+# English resources
+EN_KEY = 'en'
+EN_SHORTHAND = 'en_ewt'
+# models
+EN_MODELS_DIR = f'{TEST_WORKING_DIR}/models/{EN_SHORTHAND}_models'
+EN_MODEL_FILES = [f'{EN_MODELS_DIR}/{EN_SHORTHAND}_{model_fname}' for model_fname in MODEL_FILES]
+
+# French resources
+FR_KEY = 'fr'
+FR_SHORTHAND = 'fr_gsd'
+# regression file paths
+FR_TEST_IN = f'{TEST_WORKING_DIR}/in/fr_gsd.test.txt'
+FR_TEST_OUT = f'{TEST_WORKING_DIR}/out/fr_gsd.test.txt.out'
+FR_TEST_GOLD_OUT = f'{TEST_WORKING_DIR}/out/fr_gsd.test.txt.out.gold'
+# models
+FR_MODELS_DIR = f'{TEST_WORKING_DIR}/models/{FR_SHORTHAND}_models'
+FR_MODEL_FILES = [f'{FR_MODELS_DIR}/{FR_SHORTHAND}_{model_fname}' for model_fname in MODEL_FILES]
+
+# Other language resources
+AR_SHORTHAND = 'ar_padt'
+DE_SHORTHAND = 'de_gsd'
+KK_SHORTHAND = 'kk_ktb'
+KO_SHORTHAND = 'ko_gsd'
+
+
+# utils for clean up
+# only allow removal of dirs/files in this approved list
+REMOVABLE_PATHS = ['en_ewt_models', 'en_ewt_tokenizer.pt', 'en_ewt_mwt_expander.pt', 'en_ewt_tagger.pt',
+                   'en_ewt.pretrain.pt', 'en_ewt_lemmatizer.pt', 'en_ewt_parser.pt', 'fr_gsd_models',
+                   'fr_gsd_tokenizer.pt', 'fr_gsd_mwt_expander.pt', 'fr_gsd_tagger.pt', 'fr_gsd.pretrain.pt',
+                   'fr_gsd_lemmatizer.pt', 'fr_gsd_parser.pt', 'ar_padt_models', 'ar_padt_tokenizer.pt',
+                   'ar_padt_mwt_expander.pt', 'ar_padt_tagger.pt', 'ar_padt.pretrain.pt', 'ar_padt_lemmatizer.pt',
+                   'ar_padt_parser.pt', 'de_gsd_models', 'de_gsd_tokenizer.pt', 'de_gsd_mwt_expander.pt',
+                   'de_gsd_tagger.pt', 'de_gsd.pretrain.pt', 'de_gsd_lemmatizer.pt', 'de_gsd_parser.pt',
+                   'kk_ktb_models', 'kk_ktb_tokenizer.pt', 'kk_ktb_mwt_expander.pt', 'kk_ktb_tagger.pt',
+                   'kk_ktb.pretrain.pt', 'kk_ktb_lemmatizer.pt', 'kk_ktb_parser.pt', 'ko_gsd_models',
+                   'ko_gsd_tokenizer.pt', 'ko_gsd_mwt_expander.pt', 'ko_gsd_tagger.pt', 'ko_gsd.pretrain.pt',
+                   'ko_gsd_lemmatizer.pt', 'ko_gsd_parser.pt']
+
+
+def safe_rm(path_to_rm):
+    """
+    Safely remove a directory of files or a file
+    1.) check path exists, files are files, dirs are dirs
+    2.) only remove things on approved list REMOVABLE_PATHS
+    3.) assert no longer exists
+    """
+    # just return if path doesn't exist
+    if not os.path.exists(path_to_rm):
+        return
+    # handle directory
+    if os.path.isdir(path_to_rm):
+        files_to_rm = [f'{path_to_rm}/{fname}' for fname in os.listdir(path_to_rm)]
+        dir_to_rm = path_to_rm
+    else:
+        files_to_rm = [path_to_rm]
+        dir_to_rm = None
+    # clear out files
+    for file_to_rm in files_to_rm:
+        if os.path.isfile(file_to_rm) and os.path.basename(file_to_rm) in REMOVABLE_PATHS:
+            os.remove(file_to_rm)
+            assert not os.path.exists(file_to_rm), f'Error removing: {file_to_rm}'
+    # clear out directory
+    if dir_to_rm is not None and os.path.isdir(dir_to_rm):
+        os.rmdir(dir_to_rm)
+        assert not os.path.exists(dir_to_rm), f'Error removing: {dir_to_rm}'
+
+def compare_ignoring_whitespace(predicted, expected):
+    predicted = re.sub('[ \t]+', ' ', predicted.strip())
+    predicted = re.sub('\r\n', '\n', predicted)
+    expected = re.sub('[ \t]+', ' ', expected.strip())
+    expected = re.sub('\r\n', '\n', expected)
+    assert predicted == expected
+

stanza/stanza/tests/data/tiny_emb.txt

@@ -0,0 +1,4 @@
+3 4
+unban 1 2 3 4
+mox 5 6 7 8
+opal 9 10 11 12

stanza/stanza/tests/datasets/test_common.py

@@ -0,0 +1,76 @@
+"""
+Test conllu manipulating routines in stanza/utils/dataset/common.py
+"""
+
+import pytest
+
+
+from stanza.utils.datasets.common import maybe_add_fake_dependencies
+# from stanza.tests import *
+
+pytestmark = [pytest.mark.travis, pytest.mark.pipeline]
+
+DEPS_EXAMPLE="""
+# text = Sh'reyan's antennae are hella thicc
+1	Sh'reyan	Sh'reyan	PROPN	NNP	Number=Sing	3	nmod:poss	3:nmod:poss	SpaceAfter=No
+2	's	's	PART	POS	_	1	case	1:case	_
+3	antennae	antenna	NOUN	NNS	Number=Plur	6	nsubj	6:nsubj	_
+4	are	be	VERB	VBP	Mood=Ind|Tense=Pres|VerbForm=Fin	6	cop	6:cop	_
+5	hella	hella	ADV	RB	_	6	advmod	6:advmod	_
+6	thicc	thicc	ADJ	JJ	Degree=Pos	0	root	0:root	_
+""".strip().split("\n")
+
+
+ONLY_ROOT_EXAMPLE="""
+# text = Sh'reyan's antennae are hella thicc
+1	Sh'reyan	Sh'reyan	PROPN	NNP	Number=Sing	_	_	_	SpaceAfter=No
+2	's	's	PART	POS	_	_	_	_	_
+3	antennae	antenna	NOUN	NNS	Number=Plur	_	_	_	_
+4	are	be	VERB	VBP	Mood=Ind|Tense=Pres|VerbForm=Fin	_	_	_	_
+5	hella	hella	ADV	RB	_	_	_	_	_
+6	thicc	thicc	ADJ	JJ	Degree=Pos	0	root	0:root	_
+""".strip().split("\n")
+
+ONLY_ROOT_EXPECTED="""
+# text = Sh'reyan's antennae are hella thicc
+1	Sh'reyan	Sh'reyan	PROPN	NNP	Number=Sing	6	dep	_	SpaceAfter=No
+2	's	's	PART	POS	_	1	dep	_	_
+3	antennae	antenna	NOUN	NNS	Number=Plur	1	dep	_	_
+4	are	be	VERB	VBP	Mood=Ind|Tense=Pres|VerbForm=Fin	1	dep	_	_
+5	hella	hella	ADV	RB	_	1	dep	_	_
+6	thicc	thicc	ADJ	JJ	Degree=Pos	0	root	0:root	_
+""".strip().split("\n")
+
+NO_DEPS_EXAMPLE="""
+# text = Sh'reyan's antennae are hella thicc
+1	Sh'reyan	Sh'reyan	PROPN	NNP	Number=Sing	_	_	_	SpaceAfter=No
+2	's	's	PART	POS	_	_	_	_	_
+3	antennae	antenna	NOUN	NNS	Number=Plur	_	_	_	_
+4	are	be	VERB	VBP	Mood=Ind|Tense=Pres|VerbForm=Fin	_	_	_	_
+5	hella	hella	ADV	RB	_	_	_	_	_
+6	thicc	thicc	ADJ	JJ	Degree=Pos	_	_	_	_
+""".strip().split("\n")
+
+NO_DEPS_EXPECTED="""
+# text = Sh'reyan's antennae are hella thicc
+1	Sh'reyan	Sh'reyan	PROPN	NNP	Number=Sing	0	root	_	SpaceAfter=No
+2	's	's	PART	POS	_	1	dep	_	_
+3	antennae	antenna	NOUN	NNS	Number=Plur	1	dep	_	_
+4	are	be	VERB	VBP	Mood=Ind|Tense=Pres|VerbForm=Fin	1	dep	_	_
+5	hella	hella	ADV	RB	_	1	dep	_	_
+6	thicc	thicc	ADJ	JJ	Degree=Pos	1	dep	_	_
+""".strip().split("\n")
+
+
+def test_fake_deps_no_change():
+    result = maybe_add_fake_dependencies(DEPS_EXAMPLE)
+    assert result == DEPS_EXAMPLE
+
+def test_fake_deps_all_tokens():
+    result = maybe_add_fake_dependencies(NO_DEPS_EXAMPLE)
+    assert result == NO_DEPS_EXPECTED
+
+
+def test_fake_deps_only_root():
+    result = maybe_add_fake_dependencies(ONLY_ROOT_EXAMPLE)
+    assert result == ONLY_ROOT_EXPECTED

stanza/stanza/tests/datasets/test_vietnamese_renormalization.py

@@ -0,0 +1,35 @@
+import pytest
+import os
+
+from stanza.utils.datasets.vietnamese import renormalize
+
+pytestmark = [pytest.mark.travis, pytest.mark.pipeline]
+
+def test_replace_all():
+    text     = "SỌAmple tụy test file"
+    expected = "SOẠmple tuỵ test file"
+
+    assert renormalize.replace_all(text) == expected
+
+def test_replace_file(tmp_path):
+    text     = "SỌAmple tụy test file"
+    expected = "SOẠmple tuỵ test file"
+
+    orig = tmp_path / "orig.txt"
+    converted = tmp_path / "converted.txt"
+
+    with open(orig, "w", encoding="utf-8") as fout:
+        for i in range(10):
+            fout.write(text)
+            fout.write("\n")
+
+    renormalize.convert_file(orig, converted)
+
+    assert os.path.exists(converted)
+    with open(converted, encoding="utf-8") as fin:
+        lines = fin.readlines()
+
+    assert len(lines) == 10
+    for i in lines:
+        assert i.strip() == expected
+        

stanza/stanza/tests/depparse/test_depparse_data.py

@@ -0,0 +1,62 @@
+"""
+Test some pieces of the depparse dataloader
+"""
+import pytest
+from stanza.models.depparse.data import data_to_batches
+
+pytestmark = [pytest.mark.travis, pytest.mark.pipeline]
+
+def make_fake_data(*lengths):
+    data = []
+    for i, length in enumerate(lengths):
+        word = chr(ord('A') + i)
+        chunk = [[word] * length]
+        data.append(chunk)
+    return data
+
+def check_batches(batched_data, expected_sizes, expected_order):
+    for chunk, size in zip(batched_data, expected_sizes):
+        assert sum(len(x[0]) for x in chunk) == size
+    word_order = []
+    for chunk in batched_data:
+        for sentence in chunk:
+            word_order.append(sentence[0][0])
+    assert word_order == expected_order
+
+def test_data_to_batches_eval_mode():
+    """
+    Tests the chunking of batches in eval_mode
+
+    A few options are tested, such as whether or not to sort and the maximum sentence size
+    """
+    data = make_fake_data(1, 2, 3)
+    batched_data = data_to_batches(data, batch_size=5, eval_mode=True, sort_during_eval=True, min_length_to_batch_separately=None)
+    check_batches(batched_data[0], [5, 1], ['C', 'B', 'A'])
+
+    data = make_fake_data(1, 2, 6)
+    batched_data = data_to_batches(data, batch_size=5, eval_mode=True, sort_during_eval=True, min_length_to_batch_separately=None)
+    check_batches(batched_data[0], [6, 3], ['C', 'B', 'A'])
+
+    data = make_fake_data(3, 2, 1)
+    batched_data = data_to_batches(data, batch_size=5, eval_mode=True, sort_during_eval=True, min_length_to_batch_separately=None)
+    check_batches(batched_data[0], [5, 1], ['A', 'B', 'C'])
+
+    data = make_fake_data(3, 5, 2)
+    batched_data = data_to_batches(data, batch_size=5, eval_mode=True, sort_during_eval=True, min_length_to_batch_separately=None)
+    check_batches(batched_data[0], [5, 5], ['B', 'A', 'C'])
+
+    data = make_fake_data(3, 5, 2)
+    batched_data = data_to_batches(data, batch_size=5, eval_mode=True, sort_during_eval=False, min_length_to_batch_separately=3)
+    check_batches(batched_data[0], [3, 5, 2], ['A', 'B', 'C'])
+
+    data = make_fake_data(4, 1, 1)
+    batched_data = data_to_batches(data, batch_size=5, eval_mode=True, sort_during_eval=False, min_length_to_batch_separately=3)
+    check_batches(batched_data[0], [4, 2], ['A', 'B', 'C'])
+
+    data = make_fake_data(1, 4, 1)
+    batched_data = data_to_batches(data, batch_size=5, eval_mode=True, sort_during_eval=False, min_length_to_batch_separately=3)
+    check_batches(batched_data[0], [1, 4, 1], ['A', 'B', 'C'])
+
+if __name__ == '__main__':
+    test_data_to_batches()
+

stanza/stanza/tests/depparse/test_parser.py

@@ -0,0 +1,211 @@
+"""
+Run the tagger for a couple iterations on some fake data
+
+Uses a couple sentences of UD_English-EWT as training/dev data
+"""
+
+import os
+import pytest
+
+import torch
+
+from stanza.models import parser
+from stanza.models.common import pretrain
+from stanza.models.depparse.trainer import Trainer
+from stanza.tests import TEST_WORKING_DIR
+
+pytestmark = [pytest.mark.pipeline, pytest.mark.travis]
+
+TRAIN_DATA = """
+# sent_id = weblog-juancole.com_juancole_20051126063000_ENG_20051126_063000-0003
+# text = DPA: Iraqi authorities announced that they had busted up 3 terrorist cells operating in Baghdad.
+1	DPA	DPA	PROPN	NNP	Number=Sing	0	root	0:root	SpaceAfter=No
+2	:	:	PUNCT	:	_	1	punct	1:punct	_
+3	Iraqi	Iraqi	ADJ	JJ	Degree=Pos	4	amod	4:amod	_
+4	authorities	authority	NOUN	NNS	Number=Plur	5	nsubj	5:nsubj	_
+5	announced	announce	VERB	VBD	Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin	1	parataxis	1:parataxis	_
+6	that	that	SCONJ	IN	_	9	mark	9:mark	_
+7	they	they	PRON	PRP	Case=Nom|Number=Plur|Person=3|PronType=Prs	9	nsubj	9:nsubj	_
+8	had	have	AUX	VBD	Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin	9	aux	9:aux	_
+9	busted	bust	VERB	VBN	Tense=Past|VerbForm=Part	5	ccomp	5:ccomp	_
+10	up	up	ADP	RP	_	9	compound:prt	9:compound:prt	_
+11	3	3	NUM	CD	NumForm=Digit|NumType=Card	13	nummod	13:nummod	_
+12	terrorist	terrorist	ADJ	JJ	Degree=Pos	13	amod	13:amod	_
+13	cells	cell	NOUN	NNS	Number=Plur	9	obj	9:obj	_
+14	operating	operate	VERB	VBG	VerbForm=Ger	13	acl	13:acl	_
+15	in	in	ADP	IN	_	16	case	16:case	_
+16	Baghdad	Baghdad	PROPN	NNP	Number=Sing	14	obl	14:obl:in	SpaceAfter=No
+17	.	.	PUNCT	.	_	1	punct	1:punct	_
+
+# sent_id = weblog-juancole.com_juancole_20051126063000_ENG_20051126_063000-0004
+# text = Two of them were being run by 2 officials of the Ministry of the Interior!
+1	Two	two	NUM	CD	NumForm=Word|NumType=Card	6	nsubj:pass	6:nsubj:pass	_
+2	of	of	ADP	IN	_	3	case	3:case	_
+3	them	they	PRON	PRP	Case=Acc|Number=Plur|Person=3|PronType=Prs	1	nmod	1:nmod:of	_
+4	were	be	AUX	VBD	Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin	6	aux	6:aux	_
+5	being	be	AUX	VBG	VerbForm=Ger	6	aux:pass	6:aux:pass	_
+6	run	run	VERB	VBN	Tense=Past|VerbForm=Part|Voice=Pass	0	root	0:root	_
+7	by	by	ADP	IN	_	9	case	9:case	_
+8	2	2	NUM	CD	NumForm=Digit|NumType=Card	9	nummod	9:nummod	_
+9	officials	official	NOUN	NNS	Number=Plur	6	obl	6:obl:by	_
+10	of	of	ADP	IN	_	12	case	12:case	_
+11	the	the	DET	DT	Definite=Def|PronType=Art	12	det	12:det	_
+12	Ministry	Ministry	PROPN	NNP	Number=Sing	9	nmod	9:nmod:of	_
+13	of	of	ADP	IN	_	15	case	15:case	_
+14	the	the	DET	DT	Definite=Def|PronType=Art	15	det	15:det	_
+15	Interior	Interior	PROPN	NNP	Number=Sing	12	nmod	12:nmod:of	SpaceAfter=No
+16	!	!	PUNCT	.	_	6	punct	6:punct	_
+
+""".lstrip()
+
+
+DEV_DATA = """
+1	From	from	ADP	IN	_	3	case	3:case	_
+2	the	the	DET	DT	Definite=Def|PronType=Art	3	det	3:det	_
+3	AP	AP	PROPN	NNP	Number=Sing	4	obl	4:obl:from	_
+4	comes	come	VERB	VBZ	Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin	0	root	0:root	_
+5	this	this	DET	DT	Number=Sing|PronType=Dem	6	det	6:det	_
+6	story	story	NOUN	NN	Number=Sing	4	nsubj	4:nsubj	_
+7	:	:	PUNCT	:	_	4	punct	4:punct	_
+
+""".lstrip()
+
+
+
+class TestParser:
+    @pytest.fixture(scope="class")
+    def wordvec_pretrain_file(self):
+        return f'{TEST_WORKING_DIR}/in/tiny_emb.pt'
+
+    def run_training(self, tmp_path, wordvec_pretrain_file, train_text, dev_text, augment_nopunct=False, extra_args=None):
+        """
+        Run the training for a few iterations, load & return the model
+        """
+        train_file = str(tmp_path / "train.conllu")
+        dev_file = str(tmp_path / "dev.conllu")
+        pred_file = str(tmp_path / "pred.conllu")
+
+        save_name = "test_parser.pt"
+        save_file = str(tmp_path / save_name)
+
+        with open(train_file, "w", encoding="utf-8") as fout:
+            fout.write(train_text)
+
+        with open(dev_file, "w", encoding="utf-8") as fout:
+            fout.write(dev_text)
+
+        args = ["--wordvec_pretrain_file", wordvec_pretrain_file,
+                "--train_file", train_file,
+                "--eval_file", dev_file,
+                "--output_file", pred_file,
+                "--log_step", "10",
+                "--eval_interval", "20",
+                "--max_steps", "100",
+                "--shorthand", "en_test",
+                "--save_dir", str(tmp_path),
+                "--save_name", save_name,
+                # in case we are doing a bert test
+                "--bert_start_finetuning", "10",
+                "--bert_warmup_steps", "10",
+                "--lang", "en"]
+        if not augment_nopunct:
+            args.extend(["--augment_nopunct", "0.0"])
+        if extra_args is not None:
+            args = args + extra_args
+        trainer = parser.main(args)
+
+        assert os.path.exists(save_file)
+        pt = pretrain.Pretrain(wordvec_pretrain_file)
+        # test loading the saved model
+        saved_model = Trainer(pretrain=pt, model_file=save_file)
+        return trainer
+
+    def test_train(self, tmp_path, wordvec_pretrain_file):
+        """
+        Simple test of a few 'epochs' of tagger training
+        """
+        self.run_training(tmp_path, wordvec_pretrain_file, TRAIN_DATA, DEV_DATA)
+
+    def test_with_bert_nlayers(self, tmp_path, wordvec_pretrain_file):
+        self.run_training(tmp_path, wordvec_pretrain_file, TRAIN_DATA, DEV_DATA, extra_args=['--bert_model', 'hf-internal-testing/tiny-bert', '--bert_hidden_layers', '2'])
+
+    def test_with_bert_finetuning(self, tmp_path, wordvec_pretrain_file):
+        trainer = self.run_training(tmp_path, wordvec_pretrain_file, TRAIN_DATA, DEV_DATA, extra_args=['--bert_model', 'hf-internal-testing/tiny-bert', '--bert_finetune', '--bert_hidden_layers', '2'])
+        assert 'bert_optimizer' in trainer.optimizer.keys()
+        assert 'bert_scheduler' in trainer.scheduler.keys()
+
+    def test_with_bert_finetuning_resaved(self, tmp_path, wordvec_pretrain_file):
+        """
+        Check that if we save, then load, then save a model with a finetuned bert, that bert isn't lost
+        """
+        trainer = self.run_training(tmp_path, wordvec_pretrain_file, TRAIN_DATA, DEV_DATA, extra_args=['--bert_model', 'hf-internal-testing/tiny-bert', '--bert_finetune', '--bert_hidden_layers', '2'])
+        assert 'bert_optimizer' in trainer.optimizer.keys()
+        assert 'bert_scheduler' in trainer.scheduler.keys()
+
+        save_name = trainer.args['save_name']
+        filename = tmp_path / save_name
+        assert os.path.exists(filename)
+        checkpoint = torch.load(filename, lambda storage, loc: storage, weights_only=True)
+        assert any(x.startswith("bert_model") for x in checkpoint['model'].keys())
+
+        # Test loading the saved model, saving it, and still having bert in it
+        # even if we have set bert_finetune to False for this incarnation
+        pt = pretrain.Pretrain(wordvec_pretrain_file)
+        args = {"bert_finetune": False}
+        saved_model = Trainer(pretrain=pt, model_file=filename, args=args)
+
+        saved_model.save(filename)
+
+        # This is the part that would fail if the force_bert_saved option did not exist
+        checkpoint = torch.load(filename, lambda storage, loc: storage, weights_only=True)
+        assert any(x.startswith("bert_model") for x in checkpoint['model'].keys())
+
+    def test_with_peft(self, tmp_path, wordvec_pretrain_file):
+        trainer = self.run_training(tmp_path, wordvec_pretrain_file, TRAIN_DATA, DEV_DATA, extra_args=['--bert_model', 'hf-internal-testing/tiny-bert', '--bert_finetune', '--bert_hidden_layers', '2', '--use_peft'])
+        assert 'bert_optimizer' in trainer.optimizer.keys()
+        assert 'bert_scheduler' in trainer.scheduler.keys()
+
+    def test_single_optimizer_checkpoint(self, tmp_path, wordvec_pretrain_file):
+        trainer = self.run_training(tmp_path, wordvec_pretrain_file, TRAIN_DATA, DEV_DATA, extra_args=['--optim', 'adam'])
+
+        save_dir = trainer.args['save_dir']
+        save_name = trainer.args['save_name']
+        checkpoint_name = trainer.args["checkpoint_save_name"]
+
+        assert os.path.exists(os.path.join(save_dir, save_name))
+        assert checkpoint_name is not None
+        assert os.path.exists(checkpoint_name)
+
+        assert len(trainer.optimizer) == 1
+        for opt in trainer.optimizer.values():
+            assert isinstance(opt, torch.optim.Adam)
+
+        pt = pretrain.Pretrain(wordvec_pretrain_file)
+        checkpoint = Trainer(args=trainer.args, pretrain=pt, model_file=checkpoint_name)
+        assert checkpoint.optimizer is not None
+        assert len(checkpoint.optimizer) == 1
+        for opt in checkpoint.optimizer.values():
+            assert isinstance(opt, torch.optim.Adam)
+
+    def test_two_optimizers_checkpoint(self, tmp_path, wordvec_pretrain_file):
+        trainer = self.run_training(tmp_path, wordvec_pretrain_file, TRAIN_DATA, DEV_DATA, extra_args=['--optim', 'adam', '--second_optim', 'sgd', '--second_optim_start_step', '40'])
+
+        save_dir = trainer.args['save_dir']
+        save_name = trainer.args['save_name']
+        checkpoint_name = trainer.args["checkpoint_save_name"]
+
+        assert os.path.exists(os.path.join(save_dir, save_name))
+        assert checkpoint_name is not None
+        assert os.path.exists(checkpoint_name)
+
+        assert len(trainer.optimizer) == 1
+        for opt in trainer.optimizer.values():
+            assert isinstance(opt, torch.optim.SGD)
+
+        pt = pretrain.Pretrain(wordvec_pretrain_file)
+        checkpoint = Trainer(args=trainer.args, pretrain=pt, model_file=checkpoint_name)
+        assert checkpoint.optimizer is not None
+        assert len(checkpoint.optimizer) == 1
+        for opt in trainer.optimizer.values():
+            assert isinstance(opt, torch.optim.SGD)
+

stanza/stanza/tests/langid/test_multilingual.py

@@ -0,0 +1,104 @@
+"""
+Tests specifically for the MultilingualPipeline
+"""
+
+from collections import defaultdict
+
+import pytest
+
+from stanza.pipeline.multilingual import MultilingualPipeline
+
+from stanza.tests import TEST_MODELS_DIR
+
+pytestmark = [pytest.mark.pipeline, pytest.mark.travis]
+
+def run_multilingual_pipeline(en_has_dependencies=True, fr_has_dependencies=True, **kwargs):
+    english_text = "This is an English sentence."
+    english_words = ["This", "is", "an", "English", "sentence", "."]
+    english_deps_gold = "\n".join((
+        "('This', 5, 'nsubj')",
+        "('is', 5, 'cop')",
+        "('an', 5, 'det')",
+        "('English', 5, 'amod')",
+        "('sentence', 0, 'root')",
+        "('.', 5, 'punct')"
+    ))
+    if not en_has_dependencies:
+        english_deps_gold = ""
+
+    french_text = "C'est une phrase française."
+    french_words = ["C'", "est", "une", "phrase", "française", "."]
+    french_deps_gold = "\n".join((
+        "(\"C'\", 4, 'nsubj')",
+        "('est', 4, 'cop')",
+        "('une', 4, 'det')",
+        "('phrase', 0, 'root')",
+        "('française', 4, 'amod')",
+        "('.', 4, 'punct')"
+    ))
+    if not fr_has_dependencies:
+        french_deps_gold = ""
+
+    if 'lang_configs' in kwargs:
+        nlp = MultilingualPipeline(model_dir=TEST_MODELS_DIR, download_method=None, **kwargs)
+    else:
+        lang_configs = {"en": {"processors": "tokenize,pos,lemma,depparse"},
+                        "fr": {"processors": "tokenize,pos,lemma,depparse"}}
+        nlp = MultilingualPipeline(model_dir=TEST_MODELS_DIR, download_method=None, lang_configs=lang_configs, **kwargs)
+    docs = [english_text, french_text]
+    docs = nlp(docs)
+
+    assert docs[0].lang == "en"
+    assert len(docs[0].sentences) == 1
+    assert [x.text for x in docs[0].sentences[0].words] == english_words
+    assert docs[0].sentences[0].dependencies_string() == english_deps_gold
+
+    assert len(docs[1].sentences) == 1
+    assert docs[1].lang == "fr"
+    assert [x.text for x in docs[1].sentences[0].words] == french_words
+    assert docs[1].sentences[0].dependencies_string() == french_deps_gold
+
+
+def test_multilingual_pipeline():
+    """
+    Basic test of multilingual pipeline
+    """
+    run_multilingual_pipeline()
+
+def test_multilingual_pipeline_small_cache():
+    """
+    Test with the cache size 1
+    """
+    run_multilingual_pipeline(max_cache_size=1)
+
+
+def test_multilingual_config():
+    """
+    Test with only tokenize for the EN pipeline
+    """
+    lang_configs = {
+        "en": {"processors": "tokenize"}
+    }
+
+    run_multilingual_pipeline(en_has_dependencies=False, lang_configs=lang_configs)
+
+def test_multilingual_processors_limited():
+    """
+    Test loading an available subset of processors
+    """
+    run_multilingual_pipeline(en_has_dependencies=False, fr_has_dependencies=False, lang_configs={}, processors="tokenize")
+    run_multilingual_pipeline(en_has_dependencies=True, fr_has_dependencies=False, lang_configs={"en": {"processors": "tokenize,pos,lemma,depparse"}}, processors="tokenize")
+    # this should not fail, as it will drop the zzzzzzzzzz processor for the languages which don't have it
+    run_multilingual_pipeline(en_has_dependencies=False, fr_has_dependencies=False, lang_configs={}, processors="tokenize,zzzzzzzzzz")
+
+
+def test_defaultdict_config():
+    """
+    Test that you can pass in a defaultdict for the lang_configs argument
+    """
+    lang_configs = defaultdict(lambda: dict(processors="tokenize"))
+    run_multilingual_pipeline(en_has_dependencies=False, fr_has_dependencies=False, lang_configs=lang_configs)
+
+    lang_configs = defaultdict(lambda: dict(processors="tokenize"))
+    lang_configs["en"] = {"processors": "tokenize,pos,lemma,depparse"}
+    run_multilingual_pipeline(en_has_dependencies=True, fr_has_dependencies=False, lang_configs=lang_configs)

stanza/stanza/tests/lemma_classifier/test_training.py

@@ -0,0 +1,53 @@
+import glob
+import os
+
+import pytest
+
+pytestmark = [pytest.mark.pipeline, pytest.mark.travis]
+
+from stanza.models.lemma_classifier import train_lstm_model
+from stanza.models.lemma_classifier import train_transformer_model
+from stanza.models.lemma_classifier.base_model import LemmaClassifier
+from stanza.models.lemma_classifier.evaluate_models import evaluate_model
+
+from stanza.tests import TEST_WORKING_DIR
+from stanza.tests.lemma_classifier.test_data_preparation import convert_english_dataset
+
+@pytest.fixture(scope="module")
+def pretrain_file():
+    return f'{TEST_WORKING_DIR}/in/tiny_emb.pt'
+
+def test_train_lstm(tmp_path, pretrain_file):
+    converted_files = convert_english_dataset(tmp_path)
+
+    save_name = str(tmp_path / 'lemma.pt')
+
+    train_file = converted_files[0]
+    eval_file = converted_files[1]
+    train_args = ['--wordvec_pretrain_file', pretrain_file,
+                  '--save_name', save_name,
+                  '--train_file', train_file,
+                  '--eval_file', eval_file]
+    trainer = train_lstm_model.main(train_args)
+
+    evaluate_model(trainer.model, eval_file)
+    # test that loading the model works
+    model = LemmaClassifier.load(save_name, None)
+
+def test_train_transformer(tmp_path, pretrain_file):
+    converted_files = convert_english_dataset(tmp_path)
+
+    save_name = str(tmp_path / 'lemma.pt')
+
+    train_file = converted_files[0]
+    eval_file = converted_files[1]
+    train_args = ['--bert_model', 'hf-internal-testing/tiny-bert',
+                  '--save_name', save_name,
+                  '--train_file', train_file,
+                  '--eval_file', eval_file]
+    trainer = train_transformer_model.main(train_args)
+
+    evaluate_model(trainer.model, eval_file)
+
+    # test that loading the model works
+    model = LemmaClassifier.load(save_name, None)