ALQAC 2022

Workshop Session: Automated Legal Question Answering Competition (ALQAC 2022)

Automated Legal Question Answering Competition (ALQAC)
Run in association with the International Conference on Knowledge and Systems Engineering

ALQAC-2022 CALL FOR TASK PARTICIPATION

ALQAC-2022 Workshop: October 19-21, 2022
ALQAC-2022 Registration due: May 30, 2022

Overview

As an associated event of KSE 2022, we are happy to announce the 2nd Automated Legal Question Answering Competition (ALQAC 2022). ALQAC includes 2 tasks: (1) Legal Document Retrieval, and (2) Legal Question Answering. For the competition, we introduce the Legal Question Answering dataset – a manually annotated dataset based on well-known statute laws in the Vietnamese Language. Through the competition, we aim to develop a research community on legal support systems.

Prize

We have 2 tasks. For each task, we have

One First prize, the winning team will receive 250$.

One Second prize, the runner-up team will receive 150$.

Two Third prizes, each receives 50$.

In total, the prize is 1000$. Besides, the winning team of each task will have the KSE conference fee of the presenter covered.

Dataset

The dataset file formats are shown via examples as follows.

Legal Articles: Details about each article are in the following format:

[
    {
      "id": "45/2019/QH14",
      "articles": [
            {
                "text": "The content of legal article",
                "id": "1"
            }
        ]
    }
]

Annotation Samples: Details about each sample are in the following format:

[
    {
        "question_id": "q-1",
        "text": "The content of question or statement",
        "answer": <span of text>,
        "relevant_articles": [
            {
                "law_id": "45/2019/QH14",
                "article_id": "1"
            }
        ]
    }
]

Tasks

Tasks Description

Task 1: Legal Document Retrieval

Task 1’s goal is to return articles that are related to a statement. An article is considered “relevant” to a statement iff the statement can be answered/verified by the article.

Specifically, the input samples consist of:

Legal Articles: whose format is the same as Legal Articles described in the Dataset section.
Question: whose format is in JSON as follows:

[
    {
        "question_id": "q-1",
        "text": "The content of question or statement"
    }
]

The system should retrieve all the relevant articles as follows:

[
    {
        "question_id": "q-1",
        "text": "The content of question or statement",
        "relevant_articles": [
            {
                "law_id": "45/2019/QH14",
                "article_id": "1"
            }
        ]
    }
]

Note that “relevant_articles” are the list of all relevant articles for the questions/statements.

The evaluation methods are precision, recall, and F2-measure as follows:

Precision_i = the number of correctly retrieved articles of query i^th
the number of retrieved articles of query i^th ,

Recall_i = the number of correctly retrieved cases(paragraphs) of query i^th
the number of relevant cases(paragraphs) of query i^th ,

F2_i= (5 x Precision_i x Recall_i)
(4Precision_i + Recall_i)

F2 = average of (F2_i)

In addition to the above evaluation measures, ordinal information retrieval measures such as Mean Average Precision and R-precision can be used for discussing the characteristics of the submission results.

In ALQAC 2022, the method used to calculate the final evaluation score of all queries is macro-average (evaluation measure is calculated for each query and their average is used as the final evaluation measure) instead of micro-average (evaluation measure is calculated using results of all queries).

Task 2: Legal Question Answering

Factoid Question

Given a legal question, the goal is to produce a span of text which determines the exact answer to the question. That is, for each question-answer pair, if the character sequence of the model’s prediction exactly matches the character sequence of the true answer, the predicted answer is considered correct; otherwise, it is considered incorrect.

Specifically, the input samples consist of the question as follows:

[
    {
        "question_id": "q-1",
        "text": "The content of question",
    }
]

The system should answer whether the question/statement is a span via “answer” in JSON format as follows:

[
    {
        "question_id": "q-1",
        "text": "The content of question",
        "answer": <span of text>
    }
]

The evaluation measure will be accuracy, with respect to whether the question was correctly confirmed:

Accuracy = (the number of queries which were correctly confirmed as true or false)
(the number of all queries)

Note: The output submitted by teams will be published to a public GitHub repository so that legal and AI experts can refer to this information for analysis purposes. Expert evaluation is the official metric to decide the systems’ performance of the teams.

Submission Details

Participants are required to submit a paper on their method and experimental results. The participants have to submit via e-mail files containing the results of each task, separately. For each task, participants can submit a maximum of 3 results corresponding to 3 different settings/methods. The code for evaluation is published on Google Colab (https://colab.research.google.com/drive/17tEVE2C56kHXxdfeooBPSA5mG4vZVTxf).

In this framework, we defined the mentioned input/output data structure and evaluation methods for all 2 tasks.

Note: Participants have the responsibility to warranty their result files to follow the required format.

These examples are outputs of 2 tasks that the participants’ model needs to generate for evaluation methods:

Task 1: Legal Document Retrieval

[
    {
        "question_id": "q-193",
        "relevant_articles": [
            {
                "law_id": "100/2015/QH13",
                "article_id": "177"
            }
        ]
    },
    ...
]

Task 2: Legal Question Answering

[
    {
        "question_id": "q-193",
        "answer": <span of text>
    },
    ...
]

At least one of the authors of an accepted paper has to present the paper at the ALQAC workshop of KSE 2022.

The papers authored by the task winners will be included in the main KSE 2022 proceedings if ALQAC organizers admit the paper novelty after the review process.

Papers should conform to the standards set out on the KSE 2022 webpage (section Submission) and be submitted to EasyChair.

Application Details

Potential participants in ALQAC-2022 should respond to this call for participation by submitting an application via: tinyurl.com/ALQAC2022Registration.

Schedule (Timezone: AOE)

May 6, 2022: Call for participation
June 1, 2022: Training data release
June 20, 2022: Testing data release
August 15, 2022: Submission deadline for Task 1 & 2
August 20, 2022: Announcements of rankings/assessments
August 31, 2022: Paper/Technical report of your methods for the tasks
September 7, 2022: Notification of Acceptance
September 14, 2022: Camera-ready Submission
September 24, 2022: KSE Registration Deadline
October 19-21, 2022: KSE 2022

Task winners

– Task 1 (this task has 2 first prizes and 2 third prizes):

Team Miko

First prize

Team QA12

First prize

Team NG Team

Third prize

Team VLH

Third prize

– Task 2 (this task has 1 first prize, 1 second prize, and 1 third prize):

Team QA12

First prize

Team VLH

Second prize

Team Miko

Third prize

Questions and Further Information

Email: chau.nguyen@jaist.ac.jp with the subject [ALQAC-2022] <Content>

Program Committee

Nguyen Le Minh, Japan Advanced Institute of Science and Technology (JAIST), Japan
Tran Duc Vu, The Institute of Statistical Mathematics (ISM), Japan
Phan Viet Anh, Le Quy Don Technical University (LQDTU), Vietnam
Nguyen Minh Tien, Hung Yen University of Technology and Education (UTEHY), Vietnam
Nguyen Truong Son, Ho Chi Minh University of Science (VNU-HCMUS), Vietnam
Nguyen Tien Huy, Ho Chi Minh University of Science (VNU-HCMUS), Vietnam
Nguyen Ha Thanh, National Institute of Informatics, Japan
Bui Minh Quan, Japan Advanced Institute of Science and Technology (JAIST), Japan
Dang Tran Binh, Japan Advanced Institute of Science and Technology (JAIST), Japan
Vuong Thi Hai Yen, University of Engineering and Technology (VNU-UET), Vietnam
Nguyen Minh Phuong, Japan Advanced Institute of Science and Technology (JAIST), Japan
Nguyen Minh Chau, Japan Advanced Institute of Science and Technology (JAIST), Japan
Le Nguyen Khang, Japan Advanced Institute of Science and Technology (JAIST), Japan
Nguyen Dieu Hien, Japan Advanced Institute of Science and Technology (JAIST), Japan
Nguyen Thu Trang, Japan Advanced Institute of Science and Technology (JAIST), Japan
Do Dinh Truong, Japan Advanced Institute of Science and Technology (JAIST), Japan

Workshop Session: Automated Legal Question Answering Competition (ALQAC 2022)

ALQAC-2022 CALL FOR TASK PARTICIPATION

Sponsored by

Overview

Dataset

Tasks

Tasks Description

Task 1: Legal Document Retrieval

Task 2: Legal Question Answering

Factoid Question

Submission Details

Task 1: Legal Document Retrieval

Task 2: Legal Question Answering

Application Details

Schedule (Timezone: AOE)

Task winners

Questions and Further Information

Program Committee