Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Prompt-scarcity] List of cases when the prompt is not sufficient (feel free to add) #307

Open
Frosendroska opened this issue Jul 26, 2024 · 0 comments
Labels
improvement New feature

Comments

@Frosendroska
Copy link
Contributor

Frosendroska commented Jul 26, 2024

1. Not adding enum properties to the prompt when the enum is the part of function parameters

We want to test the following class inside the TestSpark

org.jetbrains.research.testspark.core.test.strategies.JUnitTestSuiteParserStrategy

the prompt looks like this


Generate unit tests in kotlin for org.jetbrains.research.testspark.core.test.strategies.JUnitTestSuiteParserStrategy to achieve 100% line coverage for this class.
Dont use @before and @after test methods.
Make tests as atomic as possible.
All tests should be for JUnit 5.
In case of mocking, use Mockito 5. But, do not use mocking for all tests.
Name all methods according to the template - [MethodUnderTest][Scenario]Test, and use only English letters.
The source code of class under test is as follows:

package org.jetbrains.research.testspark.core.test.strategies

import org.jetbrains.research.testspark.core.data.JUnitVersion
import org.jetbrains.research.testspark.core.test.TestCaseParseResult
import org.jetbrains.research.testspark.core.test.TestSuiteParserStrategy
import org.jetbrains.research.testspark.core.test.data.TestCaseGeneratedByLLM
import org.jetbrains.research.testspark.core.test.data.TestLine
import org.jetbrains.research.testspark.core.test.data.TestLineType
import org.jetbrains.research.testspark.core.test.data.TestSuiteGeneratedByLLM
class JUnitTestSuiteParserStrategy : TestSuiteParserStrategy {
<the full body of the class>
}

org.jetbrains.research.testspark.core.test.strategies.JUnitTestSuiteParserStrategy extends org.jetbrains.research.testspark.core.test.TestSuiteParserStrategy. The source code of org.jetbrains.research.testspark.core.test.TestSuiteParserStrategy is:

package org.jetbrains.research.testspark.core.test

import org.jetbrains.research.testspark.core.data.JUnitVersion
import org.jetbrains.research.testspark.core.test.data.TestSuiteGeneratedByLLM
interface TestSuiteParserStrategy {
    fun parseTestSuite(
        rawText: String,
        junitVersion: JUnitVersion,
        importPattern: Regex,
        packageName: String,
        testNamePattern: String,
    ): TestSuiteGeneratedByLLM?
}

Generate public classes and public methods. Response only a code with tests between `, do not provide any other text.


As you can notice, in this prompt we do not have any information about parameters in JUnitVersion, therefore generated tests will not be compilable

@Frosendroska Frosendroska added the improvement New feature label Jul 26, 2024
@Frosendroska Frosendroska self-assigned this Jul 26, 2024
@Frosendroska Frosendroska changed the title [Prompt-scarcity] List of cases when the prompt is not sufficient [Prompt-scarcity] List of cases when the prompt is not sufficient (feel free to add) Jul 26, 2024
@Frosendroska Frosendroska added the In progress PR is in progress label Jul 26, 2024
@arksap2002 arksap2002 removed the In progress PR is in progress label Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement New feature
Projects
None yet
Development

No branches or pull requests

2 participants